Deliver Free Content To App Users In 5 Easy Steps

Looking to use data from a website or 3rd party source in your app?
Parsing data in your app provides value and depth to your users. One of the biggest apps Bluecloud worked on (Grail), is a data news aggregator that delivers the most shared articles and videos from across the internet on live trending topics.  
But are app developers allowed to use someone else’s data in apps?
Keep reading to learn about what data scraping is, how to pull data off websites, when to use it + the legalities.

What Is Data Scraping?

For anyone who doesn’t know what data scraping is…

Data Scraping  (or Mining) is a technique to extract large amounts of data from websites whereby the data is extracted and saved to a local file in your computer or to a database in table (spreadsheet) format.

Simply put – you’re pulling data from a third party source (usually HTML) to use for your project.
Any data that can be viewed on a webpage can be scraped.
But can you scrape data from a mobile app?
You sure can!

Use 3rd Party Content & Scraped Data In Your Apps

One of the luxuries of being an app developer are all the data and tools available at our fingertips.
There are thousands of APIs (application programming interface) that allow app users to purchase products, make reservations, share social media posts, and even book flights all with the swipe of a finger.
And most APIs are free for developer use.
Here’s a killer app idea based off APIs:

Lets say you want to make an app to notify surfers when waves are nearby.  Lets call it Surf Alarm – Big Swell Notifier…  
You could use the Spitcast API for surf predictions and the Marine Weather API to display wave height, swell, and tides.
Notify surfers when there’s a swell nearby using the OneSignal API for Push Notifications and even create your own surf store that sells wetsuits and surf apparel in the app using the Shopify API.
Need directions to surf spots? The Google Maps API has you covered.
Surfs up!

Here are a few of my favorite APIs that can be used in your apps:

1. MailChimp API – collect user emails and store them in your MailChimp account
2. Twitter API – access to the latest Tweets
3. Facebook API – get access to Facebook’s “social graph”
4. Yelp API – access to search for over 50,000,000 businesses in 32 countries
5. Google News API – get live headlines, images, and other article metadata from Google News.

But what happens when you need data and there is no API?
Panic attack…
No worries, we’ve got you covered.

How To Scrape Data for Your App In 5 Easy Steps

If you cannot find an API that hosts the data you need for your app, follow these 5 steps:

STEP 1: Browse the internet and find the data you need.

STEP 2: If there is no API, look to see if the site supports RSS subscriptions. Look for an orange icon with a set of white lines running through it:

If there is no RSS feed, continue to Step 3.

STEP 3: Read your target site’s Terms of Use and Copyright notice to make sure you’re not going to get in trouble. Even better, contact a lawyer to do this for you.

STEP 4: Ask the source for permission to use the data.

STEP 5: Hire a Web Scraper, Data Extraction Expert, Data Analyst/Programmer, or even a Backend Database Programmer to scrape the data.  You can find dozens of these experts on Upwork.com.

And you’re good to go!

Is It Legal To Use Scraped Data In My App?

The answer you’ve eagerly been waiting for (drum roll…).
First off, I am not a legal expert nor am I advising you in any respect. The following is strictly based off my personal experience.
Because the scraping process uses pre-existing content, there are all sorts of ethical and legal boundaries to interpret.
Think about it, you are using someone else’s work for your own personal gain.  Just like with any other publication, the source of the content can pursue legal action against you.
The truth is, you’re probably not going to get in trouble for using mined data.  Most businesses have better things to do then police their websites and data.
However, if you don’t have permission to use third party data, you do run the risk of facing legal ramifications.
From my personal experience, your safest course of action is to ask each site for permission.  Don’t put yourself in hot water and be smart how you use other peoples work.

Next Steps…

The rule of thumb for using other people’s content in this day in age (ESPECIALLY when you’re trying to get an app approved), is that you need to be very creative about how you use that kind of content, otherwise you’re going to have a hard time getting approved.

Scraping is a legitimate way to access all kinds of data for your app.  But the only way you’re going to get approved is by:

1) getting consent to use the data, and

2)  adding a lot of value around the data.

If you rely and pull on other peoples content and try to become THE source…it’s not as interesting the way it was 5-10 years ago.
If it’s not your content, nobody is going to be head over heels for it because they have other sources to get that information from.
You need to make the third party content a supplement for something that is really valuable.
Always be asking yourself:

“How can I add more content and value to my app?”

Have a question about using scraped data, APIs, or RSS feeds in your app?  Drop a line in the comments section and we’ll reply.
 
See ya,
Mark Nagelmann