ARTICLE AD BOX
This station was written by: Christina Blake & Ethan Lyon.
Why Build a Custom Rank Tracking Tool?
There are a batch of large tools successful the marketplace for keyword fertile tracking – but astatine Seer we recovered that we had galore usage cases with integrated fertile information that wasn’t being addressed by those tools.
Many of our analyses propulsion ample amounts of information (think 100K, 200K, 500K+ keywords astatine a time!) for a snapshot of insights – possibly we’re analyzing contented gaps oregon trying to probe a caller enactment of concern for a client. We don’t request regular rankings for this – we request a massive, one-time information pull.
For immoderate clients, we mightiness person hunt presumption that we privation to support a pulse on, but bash we request regular data? We mightiness privation to premix up our tracking – precedence keywords daily, different keywords to assistance america recognize shifts successful the scenery play oregon monthly.
If a lawsuit was readying connected a migration oregon a large alteration successful their site, we mightiness adjacent privation to power play oregon monthly keywords to way regular truthful that we tin support a person oculus connected changes during those high-risk clip periods.
💡In this post, we’ll locomotion done however we approached gathering our ain keyword tracking instrumentality from the ground-up.
What Are the Benefits of a Custom Keyword Tracking Tool?
Flexibility
The keyword tracking tools we were utilizing didn’t springiness america the flexibility to alteration tracking frequency – squad members had to taxable ample keyword sets and past intimately show truthful we could crook disconnected tracking arsenic soon arsenic information was returned to forestall tracking 500K keywords regular for an investigation that we lone needed to tally doubly a year. If a squad subordinate forgot to crook disconnected tracking, we could person immoderate beauteous precocious costs for a precise elemental mistake.
Let’s accidental a squad subordinate wanted to taxable 50,000 hunt presumption for a snapshot analysis, and that the outgo of our aged instrumentality was ~$400/day for 50K keywords.
Using our aged tool, our squad subordinate would taxable their keywords, past each time they’d cheque to spot erstwhile the information had returned – usually wrong ~3-5 days. Then they’d request to unopen disconnected tracking instantly and download the data, oregon those keywords would proceed to incur costs.
What if 1 of that squad member’s clients had a large contented and they needed to leap successful to assistance them out? It would beryllium casual (and understandable) to hide to crook disconnected tracking if that squad subordinate had a large investigation and presumption coming up, and past a lawsuit occurrence that they needed to code connected apical of it. (Christina tin vouch for this – she’s been 1 of those squad members who forgot to crook disconnected tracking before).
Using our aged tools, those keywords would proceed incurring costs to the tune of $400/day until that squad subordinate remembered to crook them disconnected oregon if 1 of our squad members who was monitoring for large overages caught it. Now use that hazard to an full team. Human errors hap – and thing we could bash to assistance our squad is taking distant the work of remembering to crook disconnected keywords.
When analyzing snapshot (one-time rankings) keywords from our aged tool, we recovered that we were tracking (and paying for) 4-5x the rankings we really needed, chiefly owed to the information that information sometimes took respective days to instrumentality and sometimes squad members didn’t crook disconnected tracking immediately. Approximately 80% of those rankings were redundant.
| Year | Count of snapshot keywords | Count of deduplicated snapshot keywords | % of waste |
| 2019 | 12,059,428 | 2,461,270 | 79.59% |
| 2020 | 5,276,197 | 1,083,009 | 79.47% |
With our ain keyword tracking tool, that aforesaid squad subordinate submits their 50K hunt presumption utilizing the “one-time” frequence and that’s it. When their information is available, they tin usage it – nary other steps needed.
We’re besides capable to taxable much keywords by partnering with Traject Data than ever earlier – we went from 26M rankings successful 2019 to 61M rankings successful 2020, the twelvemonth we launched our interior fertile tracking tool.
The time this blog station was written (2/10/22) we had 4.6 cardinal rows of keyword information successful our fertile tracking information lake.
Of each of the processed rankings with Traject, we spot the pursuing frequencies:
| Frequency | Count of tracked keywords | % to total |
| Daily | 8,420,170 | 7.89% |
| Weekly | 8,955,622 | 8.39% |
| Monthly | 2,157,943 | 2.02% |
| Quarterly | 586,760 | 0.55% |
| One-Time | 86,578,430 | 81.14% |
That’s implicit 80% of each full hunt presumption tracked astatine a one-time frequence successful the past 2 years! Imagine each of the clip saved and outgo overages prevented by giving users the flexibility to way astatine antithetic frequencies.
Accessibility
When we were inactive utilizing different fertile tracking tools, we’d person to export the information if we wanted to customize thing extracurricular of their software. Even if we built templates successful visualization tools similar Power BI oregon Google Data Studio to velocity up physique time, those templates would expect precise circumstantial inputs – if a instrumentality changed the sanction of a file oregon if a squad subordinate pursuing a database of steps to export information missed a step, it could origin errors and confusion.
For example, a squad subordinate performing an investigation mightiness travel 10-20 instructions for exporting the information a circumstantial mode – possibly it calls for a circumstantial study oregon filtering the information a definite mode earlier exporting. The squad subordinate exports their CSV to their computer, opens up the template, and selects their CSV arsenic a source.
All of the sudden, everything breaks and they are deed with a glaring mistake connection – “The file “Landing Page” wasn’t found.” All of the different information sources successful the investigation neglect arsenic good – owed to the nonaccomplishment from the missing “Landing Page” column.
The squad subordinate isn’t definite what went incorrect – aft all, they followed each of the steps! They leap into a chat and inquire immoderate different squad members for help. One squad subordinate asks for a screenshot of the mistake message. Another squad subordinate says they tin leap connected a video chat to assistance troubleshoot.
Finally, idiosyncratic who ran into this occupation earlier asks our archetypal squad subordinate to unfastened up the CSV they downloaded. The “Landing Page” file isn’t successful the CSV – it has been replaced by the tract sanction “URL”.
“Sometimes this instrumentality conscionable changes file names and we don’t find retired until it breaks thing – I’ll pat the idiosyncratic who owns the template and instructions to marque an update” our much seasoned squad subordinate says.
Problem solved, but this process took respective squad members possibly 30 minutes to lick it. Our archetypal squad subordinate didn’t privation to discarded anyone’s clip – they mightiness person spent an hr trying to hole it themselves earlier they adjacent asked for help.
By controlling our information we guarantee that changes similar that don’t hap to squad members. Even if a vendor makes a change, we tin “hide” those types of changes from the squad successful our translation furniture – possibly renaming the “URL” file backmost to “Landing Page” successful a cleaning measurement earlier that information gets into the hands of our team.
Reusability
Keeping our information successful our warehouse besides gives america the accidental to reuse data. We mightiness person aggregate products that person a information root successful communal – if those could usage the aforesaid data, we could make much worth without expanding costs.
We tin besides multiply those savings done creating queues to microservices that deduplicate and cache information – decreasing costs and expanding turnaround times for squad members to get data.
Our information tin travel into scalable dashboards and reports for insights that each SEO squad subordinate wants to cognize (like “how has my fertile for precedence keywords changed week implicit week?” oregon “how did our fertile amended aft implementing that contented audit?”) but it tin besides beryllium utilized successful customized analyses by our analytics teams wherever we articulation successful a client’s paid hunt oregon CRM data.
After lone a fewer months of launching our interior fertile tracking instrumentality we started to deed information size limitations successful our visualization tool, and with our information increasing exponentially we had to determination to a information level that could query petabytes vs gigabytes. We were capable to springiness ourselves immoderate runway by implementing incremental refreshes oregon by filtering retired information that wasn’t a must-have successful each dashboard (which caused further clip for gathering each information product).
At the extremity of 2021, we migrated our information products (including fertile tracking data) from 1 information level to our ain web exertion powered by Looker’s embedded analytics. 
Because the information was successful our warehouse, we were capable to alteration the information and rebuild utilizing champion practices for our caller information platform. We kept our aged level moving until our caller level was acceptable to motorboat – thing that mightiness not person been imaginable without the quality to usage the aforesaid information successful aggregate applications.
Security
A large worth of creating our ain fertile tracking instrumentality is the quality to physique robust information into the system, not lone by keeping our information harmless successful our warehouse but by using permissions to make a amended experience for our squad members.
By joining our client’s integrated ranking information with information from our CRM (like “which squad subordinate is assigned to what client”) we tin usage permissioning to guarantee that lawsuit selling information is lone disposable to squad members moving connected that client. For a Seer squad member, erstwhile they unfastened up 1 of our information products, they lone spot their clients’ data, making navigating done products easier.
This besides gives america opportunities to anonymize information and harvester it for industry-level trends and insights, without relying connected manual methods much prone to user-error.
Democratization
Having our ain fertile tracking instrumentality means that each of that information flows into our information warehouse, wherever we tin nonstop it galore antithetic ways for our squad to devour it – we tin democratize our information by enabling our squad members to marque data-informed decisions and consciousness assured astir data, careless of their method expertise.
Remember our squad subordinate with the CSV issue?
They besides had that information stored locally successful a CSV connected their computer. If different squad subordinate wanted to physique a antithetic investigation with the aforesaid data, they’d person to spell done those steps each implicit again oregon inquire the archetypal squad subordinate to email them their CSV. All of that enactment and information is decentralized.
Storing information successful our warehouse gives america the flexibility to assistance squad members entree to the aforesaid centralized information successful antithetic formats. Data products that standard to the full squad mightiness usage information lakes, but we besides make bite-sized tables and curated views that could beryllium visualized successful Power BI, Google Data Studio, Tableau, oregon immoderate different instrumentality a squad subordinate mightiness privation to usage – we don’t unit them into circumstantial instrumentality but promote squad members to usage immoderate tools they are assured using.
This besides cuts down connected grooming and physique clip – for reports that each squad subordinate careless of acquisition oregon relation should person entree to, we’ll make a dashboard that they conscionable click to open. They don’t request to download the data, cleanable and alteration it, and past physique visualizations connected apical of it. They conscionable login and unfastened the dashboard and voila — it’s there!
For squad members who physique customized dashboards arsenic portion of their role, we’ll supply them with structured information that tin assistance them rapidly physique the instauration of their analysis, and past they usage their skills to customize it.
How Has Building Our Own Rank Tracking Product Changed How We Work?
In 2019, our squad obtained 24M rankings astatine a regular frequency. We tracked 12M snapshot rankings, but ~80% were redundant (only 2.4M unsocial rankings).
In the archetypal fractional of 2020, we obtained 14M rankings astatine a regular frequence and different 5M snapshot rankings (again, ~80% of snapshot rankings were redundant – we lone needed 1M). After we migrated to our fertile tracking instrumentality successful July 2020, we processed 46M full rankings across aggregate frequencies.
In 2021, we returned 56M rankings. 76% of full rankings were tally connected a one-time frequency. Only 9% of those keywords were tracked daily.
We’re lone a fewer weeks into 2022 and we’ve returned 4.6M rankings truthful far, with 65% of rankings tracked astatine a one-time frequency. We tin process up to 1M keywords per day, and that information flows into aggregate products and tools crossed our centralized platform.
And not a azygous squad subordinate is doing regular checks to spot if their information was returned truthful that they tin crook disconnected tracking.
Want to Work With Bigger Data?
If you’re a natural-born advisor who loves solving problems astatine standard utilizing large information — explore Seer Careers & use present (we’re hiring folks conscionable similar you!)
.png)
English (US)