Debug missing images in your service

Look up your frontend's external IP address:

$
kubectl get services
NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)        AGE
add-service            ClusterIP      10.109.114.34    <none>           80/TCP         20m
admin-service          ClusterIP      10.110.29.145    <none>           80/TCP         20m
fetch-service          ClusterIP      10.104.224.242   <none>           80/TCP         20m
frontend-service       LoadBalancer   10.102.82.89     10.102.82.89     80:32161/TCP   20m
gateway-service        LoadBalancer   10.101.237.225   10.101.237.225   80:32469/TCP   20m
kubernetes             ClusterIP      10.96.0.1        <none>           443/TCP        20m
manipulation-service   ClusterIP      10.107.23.237    <none>           80/TCP         20m
moderate-service       ClusterIP      10.105.207.153   <none>           80/TCP         20m
mysql                  ClusterIP      10.97.194.23     <none>           3306/TCP       20m
upload-service         ClusterIP      10.108.113.235   <none>           80/TCP         20m

Paste the IP in your browser:

This looks the same as it did before, except that there's a new hat style, called PIXIE.

Select a number of hats, the PIXIE style, and Hat me:

Oops! Where did Bob go? Instead of rendering Bob Ross with your hat selection, the frontend served no image at all.

Observe a network request in your browser's developer tools:

Here, the response from the image request says, "This hat style does not exist! If you want this style - try submitting it."

You know that the PIXIE hat style most certainly does exist because you chose it from the selector. But for some reason, the application can't render the image.

For good measure, try to render a different hat style:

It worked! Your users were right. There's something wrong with your application.

From the New Relic homepage, go to Kubernetes:

Choose your tiny-hat cluster:

Then click Live Debugging with Pixie:

This is Pixie's live, code-level debugger:

You use it to drill down and learn more about the services in your cluster.

Notice the script dropdown menu at the top of the debugger:

Pixie's live debugger renders data based on open source scripts written in PxL, Pixie's proprietary query language. The default script is px/cluster, which shows cluster-level information including:

• A service map
• Nodes
• Namespaces
• Services
• Pods

Scroll down to see the error rates for your services:

Yikes! You have three services returning a high percentage of errors:

• fetch-service
• frontend-service
• gateway-service

You know that the website lives at frontend-service. You can reasonably rule this out as the culprit because you know it renders other hats just fine. That leaves two potential problem services:

• gateway-service
• fetch-service

To decide which service to look at first, scroll up to NAMESPACES and choose default:

Your app services live in the default namespace. This helps you filter the service map to show more useful nodes.

Scroll up to the service map:

Here, you see that the frontend-service requests data from gateway-service. In turn, gateway-service requests data from fetch-service. So, following that order of operations, focus first on the gateway-service.

Double click the gateway-service in your service map to learn more about it:

Notice that Pixie's live debugger has seamlessly replaced namespace data with service data by changing the script:

You're now using the px/service script filtered down to default/gateway-service. In this service, you see a graph with the errors, but not much about where those errors are coming from.

Click the script selector to switch to the px/service_stats script. Filter the svc to default/gateway-service:

This gives a better picture of the errors in your service.

Scroll down to the Incoming Traffic and Outgoing Traffic tables:

The gateway service is returning the same amount of errors on inbound requests as it's receiving from outbound requests to the fetch service. This is a good indicator that you need to look at what's happening upstream.

Switch to the px/http_data_filtered script, targeting default/fetch-service and requests with a 400 response status code:

Click on a row to learn more about a request to the fetch service that resulted in an error:

Here, you see that the request path looks like /fetch?style=PIXIE&number=1. This looks right, because the hat style you chose is called PIXIE. So if the fetch service is still returning 400s, something wrong is happening when it tries to find the hat.

Switch to the px/mysql_data script and add a source filter for default/fetch-service:

Many of these queries returned no results.

Click on one with no results, and look at the req_body to see the query:

SELECT * FROM main.images WHERE BINARY description='pixie' AND approve='true'

There's the problem! The BINARY type cast effectively makes the WHERE condition case sensitive. Since the hat's style is called PIXIE, this condition fails to find it. Now that you know, you can fix this query in your fetch service.

Summary

To recap, you observed an error in your application and used Pixie in New Relic to:

1. Understand your services' relationships
2. Review the error percentages for each of your services
3. Look at individual response bodies
4. Find a semantic error in a query within one of those services

And you didn't even need to individually install agents in any of your services. Pixie was able to deliver all the information you needed!