Navigating the Limits of the Instapaper API: Exporting a Decade of Reading

Lately, I’ve been dedicating my evenings and weekends to experimenting with AI tools, revisiting old projects, and exploring new ideas. With powerful tools like Cursor and large language models such as Gemini, Claude, and ChatGPT, it’s become remarkably easy to rapidly and efficiently spin up new projects. These technologies have opened exciting avenues for creative problem-solving, and I’ve been trying to think of novel ways to leverage them.

One idea that struck me was analyzing my reading history over time. Having accumulated over 7,300 articles in my Instapaper archive over the past 13 years, I saw an opportunity to dive into this data to uncover interesting patterns. My goal involves two main steps:

1. Creating a local repository of my entire reading history in full text.
2. Leveraging AI tools to develop various analytical methods, such as sentiment analysis over time, tracking topic interests, and correlating article topics with historical world events.

However, I’ve encountered significant challenges with the Instapaper API.

Key API Limitations

After thorough testing, I’ve identified several critical constraints:

1. 500-Item Limit per Folder The Instapaper API only returns the first 500 items per folder, regardless of pagination methods.

2. Pagination Doesn’t Work as Expected Although the API documentation describes pagination options (like the have parameter), attempts to paginate beyond the initial 500 items fail, consistently returning empty responses.

3. Inconsistent Response Formats API responses differ from documentation, frequently lacking keys like bookmarks and a since cursor, notably within Archive and Starred folders.

4. Cursor and Timestamp-based Pagination Failures Multiple pagination methods—including using the since cursor, before timestamps, and skip offsets—fail, consistently returning only the first 500 articles.

Historical Context: API Access Considerations

Delving deeper, I found a 2011 blog post by Instapaper creator Marco Arment discussing concerns over the costs and revenue impact of offering full API access, leading to limitations for non-premium users. While understandable at the time, it’s unclear if the current 500-item restriction is related to these earlier decisions or driven by other factors. Nevertheless, it significantly limits programmatic access to larger collections like mine.

Practical Implications

Due to these constraints:

• Only the most recent 500 articles per folder are accessible via the API.
• There seems to be no direct API workaround for accessing larger collections.

Reaching Out to Instapaper Support

I’ve already reached out to Instapaper Support to clarify these limitations and explore potential solutions. Here’s the core of my inquiry:

I’m a premium Instapaper subscriber and developer working on a project to export my complete archive of saved articles (approximately 7,320 items) to Markdown files for personal use.

After extensive testing with the Instapaper Full API, I’ve encountered what appears to be a hard limit of 500 items per folder, even with an active premium subscription. I’ve attempted multiple approaches to access my complete archive:

1. Standard pagination using the ‘have’ parameter as documented
2. Various combinations of limit values (always ≤ 500)
3. Date-range filtering using ‘from’ and ‘to’ parameters
4. Offset-based pagination using the ‘skip’ parameter

In all cases, the API consistently returns only the 500 most recent articles from my archive, regardless of the parameters used.

I value Instapaper as a service and would appreciate any guidance on whether complete archive access via the API is possible.

I’m currently awaiting their response and guidance regarding whether complete archive access via the API is achievable.

Potential Workarounds

In the meantime, here are some potential paths forward:

1. Wait for Response from Instapaper Support Their insights might reveal new methods or clarify intentional API restrictions.

2. Explore Web Scraping While this could potentially work, it introduces complexity and risks violating Instapaper’s terms of service.

3. Adopt a Multi-Folder Strategy Restructuring bookmarks into multiple smaller folders, each under 500 items, might circumvent the API limitations.

4. Use Third-Party Export Tools Tools like instapexport might offer partial relief, but typically face similar API restrictions.

Technical Details

Here’s the specific behavior encountered with various pagination attempts:

- Using ‘have’ parameter:
    - First request: 500 bookmarks returned
    - Second request: Empty list
- Using ‘before’ timestamps:
    - First request: 500 bookmarks returned
    - Second request: Same 500 bookmarks again
- Using offset/skip:
    - First request: 500 bookmarks returned
- Second request: Same 500 bookmarks again

Clearly, the API design intentionally limits responses to the latest 500 items.

Next Steps

Despite these obstacles, I’m determined to solve this challenge. Using AI tools like Cursor, Gemini, Claude, and ChatGPT, my plan is to:

1. Develop a reliable method to access and store my complete reading history locally.
2. Utilize AI-driven analysis to extract insights from my data, exploring trends, sentiment, and correlations.
3. Share my findings, offering unique insights into how my interests have evolved over the past decade.

If you’ve faced similar issues or have creative ideas for overcoming these limitations, I’d love to hear from you!