October 10, 2018

Personalised concert recommendations using Songkick and Apify Actors

I’ve recently directed my automation addiction towards music events.

Loads of artists come through Sydney but Songkick only sends notifications for big-ticket concerts, and quite infrequently at that. What about all of the smaller gigs happening every week? Are there any recommended artists I should check out?

To fill this gap I’ve used the Songkick API to produce an RSS feed that tells me if any artists I track, or artists that are similar/recommended, are coming to town.

RSS feed of recommended Songkick events viewed in Feedly

The list of “Recommended through” artists shows which of my tracked artists they’re similar to. Sometimes it can be 10 or so artists long, which is usually a good sign I ought to be interested.

Combined with this awesome iOS 12 shortcut for looking up Spotify artists, it’s a pretty sweet workflow.

Open Spotify Artist - iOS 12 Shortcut

It’s fully automated in the cloud, for free, using Apify Actors and Crawlers. You can view the code on GitHub and read on below for how it works and how to set up your own.

How it works

For simple automation tasks I often turn to Zapier, but I’ve recently been playing with Apify and enjoying it a lot. I wrote about crawlers in my last post, but Apify also has Actors for complete flexibility.

You point the actor at a GitHub repo with a Dockerfile and some code and it will run your Docker container automatically, serverless-style. I’ve often wanted a reliable place (read: non-laptop) to automate simple scripts, and this suits my needs well.

Here is how I combined Apify’s Actors, Key-value Store, and a Crawler to produce the final RSS feed.

Apify Actors and a Crawler pulling from the Songkick API and producing an RSS feed

The following JavaScript routines carry out the necessary API calls and persistence steps:

1. One actor pulls all of my tracked artists from the Songkick API, and for each of those fetches 50 recommended artists. The results are persisted to a Key-value Store in Apify. I only run this once a month because it takes a while and my tracked artists don’t change that often.
2. Another actor pulls all Songkick events for my “metro areas” and saves any that include the artists I pulled in step 1, again to a Key-value Store. I run this every night.
3. A Crawler reads the JSON results from the Key-value Store and produces an RSS item for each event.*

How to set it up for yourself

The README on GitHub has instructions for how to set this up in Apify.

Learnings along the way

The following tips might be useful for your own actors.

Splitting out the routines into different actors

It didn’t make sense to put everything in one actor, so I split them into actors I could schedule separately. Key-value stores were useful for transferring state between different stages.

I also wanted to share some code across different actors, and my initial plan didn’t work out… Apify lets you point each actor at a subdirectory in the git repo to look for a Dockerfile, so I initially went for something this:

# This doesn't work
src/
    common_utils.js
    actor_1/
        Dockerfile
        act.js  # Uses ../common_utils.js
    actor_2/
        Dockerfile
        act.js  # Uses ../common_utils.js

But, with the way Apify runs your Dockerfile, you can’t access parent directories to copy them into your container. To solve this, all actors in the repo now share a Dockerfile which calls a single entry point script. The script checks an environment variable (you can set this per actor in the Apify UI) to work out which actor function it should run each time.

# This does work
Dockerfile  # Now all JavaScript is underneath the Dockerfile
src/
    entry_point.js  # Checks an env variable to run the right act
    common_utils.js
    actor_1/
        act.js
    actor_2/
        act.js

Installing package.json’s devDependencies in the Docker container

My code requires compilation with Babel, which meant I needed to install devDependencies from package.json when building the Docker container. I found that a simple npm install was skipping my devDependencies because the NODE_ENV environment variable was being set to production, I’m guessing by Apify. My Dockerfile therefore sets NODE_ENV=development right before using Babel, and then switches it back to production afterwards.

Future ideas

The set-up works pretty well for me at the moment but I have some additions I’d like to make at some point:

1. Add images, or links to YouTube/Spotify. This would be a nice to have—Feedly already adds a cover image automatically, and I can look up the artist in various apps pretty quickly using iOS 12 shortcuts.
2. Render the RSS content in HTML (currently it’s plain text), to add more context in a clean way. This isn’t actually supported by Apify’s crawlers (they don’t support CDATA elements), but I could achieve it via an actor where you’re allowed to produce XML with a specific content-type.

Wrapping up

You can find the code on GitHub as well as some instructions for how to set it up for yourself. Improvements are welcome!

Songkick Apify RSS Feedly

---

Previous post

Create an RSS feed from any website using Apify

Next post

How does Monaco Editor enable text editing on a web page?