Convert API Log Data into Actionable Information

You’ve built an API to solve technical problems, but you know that’s just the beginning. In addition to helping developers use it, you need to understand how they use it. You want to measure its performance and popularity, and make adjustments based on what you discover.

Maybe some developers are seeing a lot of errors when making API calls or it’s taking too long for them to get to the first “Hello World.” Perhaps the number of developers converting to paying customers is below your expectations. You want to understand the usage of your API and ensure that customers use your API in the long term.

At this point, you might reasonably look to your API logs. You’re seeking deep insights into your API and its users, but by default what you’ll find is raw usage. How can you visualize not just the usage, but also insights that help you take action? This post will cover how you might approach this on your own, as well as how Moesif can help you move more quickly in your API program.

Your API Logs Contain a Wealth of Actionable Information

  • What are the request and response payloads being sent?
  • Which API version is used the most?
  • Where is the API experiencing higher latency?
  • What is the sequence of transactions leading to errors on POST /*/count/events?
  • Which SDKs are used to access the API the most?

You can also use API log data to answer questions about the users of your API, like:

  • What are the top endpoints used by each of my customers?
  • Who are my top customers by API usage?
  • Which users are scraping large amounts of data from the API?
  • Which customers are running into 401 unauthorized errors?
  • What is the API activity for a specific customer?

Your API logs contain valuable information that you can use to the benefit of your customers and your business. But how do you access and analyze that log data?

The first tool many API product development teams turn to for API logging and metrics is Kibana.

You Probably Pipe Your Logs into Kibana

Like any stack, you’ll need to put the pieces together. While they work well together, these are three separate tools also used apart. You’ll need to stitch together the customer journey data, from UI to API. These traces, as they’re called in Kibana terms, are important for turning data into something actionable. It’s not as simple as connecting logs and being done — you need to determine what’s needed, handle collisions, and assume the ongoing cost of growing indices.

Kibana is a log visualization tool, which makes it an appropriate place to start when thinking about API analytics. However, the primary use case for Kibana is infrastructure monitoring and metrics. It’s not designed specifically for API products, so there are some drawbacks to using Kibana for API logs and metrics.

Pros and Cons of Kibana for API Logs and Metrics

Kibana Pros

The pros of Kibana include:

  • Open source and free to use.
  • Great at visualizing API logs.
  • You can explore massive volumes of log data.
  • Many useful features. Although it should be noted that some features are available separately and some are currently experimental or in beta.
  • Layered on top of Elasticsearch (also a con), making it ideal for use on high-cardinality, high-dimension log data- a must-have feature for API logs.

And now onto the cons.

Kibana Cons

Kibana’s cons include:

  • Compatible with Elasticsearch and Logstash only. If you want to use Kibana with other databases, you’re out of luck.
  • Designed for infrastructure metrics and not specifically for API products. You have to customize Kibana for API log use cases.
  • To connect user behavior and API activity into a single journey is incredibly manual and prone to errors over time.
  • You need to use the Kibana Query Language (KQL) or Lucene query syntax (Kibana legacy query language) for queries, so there is a moderate to high learning curve.
  • Maintaining the Elastic Stack takes a lot of effort. For example, you must periodically upgrade each part of the stack and make sure upgrades won’t break any plugins you’re using or require that you rewrite any of your visualizations.

Visualization is only a starting point when it comes to gaining insights from your API logs. And while Kibana is great at visualizing logs, it often leaves you wondering, “what’s next?” What action should I take based on this data?

Move Beyond Kibana and Get Actionable Insights from Your API Logs with Moesif

With Moesif, you can filter and aggregate billions of API calls and user actions on pretty much any field, even high cardinality fields like a session token or user id.

The Moesif platform includes many features and capabilities that you won’t find in Kibana and Elastic Stack, such as:

  • High-cardinality, high-dimension API metrics that are compatible with any database, not just Elasticsearch.
  • Automatic analysis of REST and GraphQL APIs.
  • Automatic insights on query parameters and HTTP text payloads like JSON and XML.
  • Track API calls, user actions and behavior, and website activity.
  • Embeddable API logs and charts.

For more details, see the complete comparison of Moesif vs. Kibana.

Moesif lets you to obtain valuable information from your API logs and helps you understand what your data is telling you through custom and pre-built dashboards.

Understand What Your API Log Data is Telling You

For example, a fintech company (and Moesif customer) created a dashboard that lets them track key API and user metrics, which includes:

  • Recent API errors
  • HTTP status requests
  • Product usage
  • Daily active users (DAU)
  • Most active users

Through the dashboard, the fintech API product team sees who their customers are, how they integrated the API with applications, and what users of their API are experiencing. The team also gains insights into how well their API is performing and how many developers moved from a trial or sandbox version of the API to a production version.

Don’t Miss Out on Valuable Insights

Want to learn more about how Moesif can help you gain deep insights from your API logs?

Get started building great APIs with Moesif API Analytics. Learn more.

Originally published at https://www.moesif.com on January 28, 2021.

With APIs and people, anything is possible. Mostly it’s the people.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store