ARTICLE AD BOX
Last updated connected June 20, 2022.
In this guide, you volition larn measurement by measurement however to usage JetOctopus to analyse log files and larn astir however Googlebot and different bots crawl your website. This is simply a basal usher that volition beryllium perfect for radical who privation to get started with log record investigation and don’t person a batch of acquisition successful the subject.
A log record investigation is 1 of the much precocious method SEO tasks which whitethorn look a spot daunting if you person ne'er done that before.
However, my extremity with this usher is to amusement you that log record investigation tin really beryllium amusive and you don’t request to beryllium an implicit pro to bash it.
I curse that – erstwhile you commencement diving heavy into log files – you volition autumn successful emotion with it.
Ready? Let’s get started.
What you request to get started with log record analysis
To beryllium capable to bash a log record investigation of your website, you request the following:
- Access to JetOctopus wherever you person your website added arsenic a project
- Access to your server to beryllium capable to download log files
- Depending connected the method you privation to entree logs, you whitethorn request an SFTP lawsuit similar Filezilla to download log files to your machine and past upload them to JetOctopus.
- Actual log files from your server
❗Make definite to besides cheque my usher connected how to audit your website with JetOctopus.
Get prepared to analyse log files
Here is each you request to bash measurement by step. My presumption is that you have conscionable created a JetOctopus account and person nary information caller there.
Prepare JetOctopus
Let’s commencement by mounting up JetOctopus and crawling your website.
1. Open JetOctopus and click + Create Project to adhd your website. Add each information arsenic required.
2. Click + New Crawl to initiate the crawl and person the freshest data. Configure the crawl settings and integrate your website with GSC (you volition request that successful the future).
3. Start the crawl and get backmost erstwhile it’s done.
4. Once the crawl is complete, navigate to Log Analyzer > Manage Logs > Integrations and enable Logs integration.
Access & download log files from your server
Now we request to really provender JetOctopus with log files. All you request to bash is entree your server and download log files from the desired period.
Depending connected your web host, this process whitethorn differ.
❗Below I’m showing you however I did it with Cloudways which I honestly deliberation is the champion hosting enactment you tin get.
1. When successful Cloudways, navigate to Applications and click connected your website.
2. Under Application Management, navigate to Access Details.
3. Under APPLICATION CREDENTIALS, you volition find the credentials for SFTP oregon SSH entree to your website. Create caller credentials if needed.
4. Open an SFTP lawsuit like Filezilla (I americium utilizing Filezilla).
5. Enter your credentials, link to your server and navigate to the logs folder.
6. Download the full folder to your computer. Review the files downloaded and prime which files you really privation to analyze.
Upload log files to JetOctopus
Let’s present upload the downloaded log files to JetOctopus:
1. When successful JetOctopus, navigate to Log Analyzer > Manage Logs > Upload files and upload the log files successful the Apache and/or Nginx format.
Your downloaded files are apt to beryllium successful .gz format. Extract them earlier uploading.
2. Upload the files and click Import Files.
3 Once the import is implicit and JetOctopus has been capable to correctly place the files, you volition spot thing similar this. Click connected Manage logs to start!
4. I person uploaded the entree log for the past 4 days (June 16-June 19).
Now you person each you request to do a log record analysis.
Analyze your log files with JetOctopus!
And present the amusive portion you person been waiting for begins!
Let maine amusement you a fewer things you tin analyse here. Of course, what precisely you privation to analyse depends connected the intent of your log record investigation and your method skills.
Here I americium showing you basal things you tin larn astir your website and the bots visiting it.
Get a wide overview of bots visiting your website.
Navigate to Overview and prime the play you privation to analyze.
As you tin spot successful the screenshot above, Googlebot accounts for 77% of bot visits to my website successful the past 48 hours.
The default segments successful Overview fto you analyse each pages visited by bots (All Pages), each the indexable pages visited by bots (All Pages), pages visited by Googlebot lone (Visited by Googlebot), visited by bots different than Googlebot (Not Visited by Googlebot), and more.
Note that you tin besides make your ain segments.
Check absorbing stats of the enactment of the main bots (Google, Bing, Yandex) connected your website.
Go to Impact and prime the bot you privation to analyze.
You tin spot the information similar what parts of your website a fixed crawler crawls the astir (i.e. the pages that are internally linked oregon not), however heavy the crawler goes, and more.
Check for imaginable wellness issues.
Navigate to Health. You volition spot the information similar types of problems encountered, dynamics of problems (bots and integrated visits), and HTTP methods.
As you tin spot successful the screenshot below, the main contented JetOctopus detected for my tract is that I person a clump of non-permanent redirects (302).
Check what different bots are visiting your site.
Navigate to Other Bots. This study is truly interesting!
You volition spot the ratio of bully vs fake bots. You tin besides spot the database of non-search bots (e.g. LinkedIn bot, Ahrefs, etc.). It is bully to cognize who is spidering your website.
Analyze GoogleBot & look for imaginable issues
And, of course, since Google is the ascendant player, we are astir funny successful what GoogleBot is doing connected our websites. Below are a fewer basal things you tin larn astir GoogleBot crawling your site.
Check what pages and resources GoogleBot is visiting.
Navigate to Bot Dynamics > Google to get an overview of however GoogleBot was visiting your website. Select the clip play you privation to analyze.
On the archetypal screen, you tin spot the accusation astir the full fig of visits, the pages visited, an overview of presumption codes, and bot load time.
Scroll down to spot much accusation similar what benignant of GoogleBot was ascendant (Desktop oregon Mobile), dynamics of mobile/desktop bots, GoogleBot versions, and more.
Click connected View pages under Crawl Budget successful Pages to spot precisely what pages GoogleBot was visiting.
This volition open Pages successful Logs section with a filter for GoogleBot applied.
Click connected View pages under Crawl Budget successful Visits to spot each resources that GoogleBot has been crawling and utilizing crawl fund for.
This volition open Raw Logs data.
Check resources that returned presumption codes different than 200 erstwhile GoogleBot visited them.
Under Data Tables > Raw Logs, select Non-200 Status to analyse presumption codes different than 200 encountered by GoogleBot.
As you tin spot successful the screenshot below, GoogleBot came crossed the 304 (Not Changed) presumption a lot.
To presumption lone pages with presumption codes different than 200, spell to Pages successful Logs > Pages with 4xx, 5xx, oregon Pages with 304.
Check if the Google crawl fund is being wasted.
Crawl fund is not an contented for tiny websites (under tens of thousands of pages) but tin truly go a immense occupation for precise ample sites (millions of pages).
You tin cheque if the crawl fund is being wasted by analyzing the pages returning presumption 4xx (error) and 5xx (server error). The second whitethorn bespeak that your server is being overloaded which whitethorn origin Google to crawl your website less.
To cheque if GoogleBot is crawling immoderate resources returning errors (4xx presumption codes), spell to Data Tables > Raw Logs > 4xx Status. If you spot thousands oregon tens of thousands of URLs returning 4xx, marque definite to analyse it much carefully.
To cheque if GoogleBot is encountering resources returning presumption 5xx, navigate to Data Tables > Raw Logs > 5xx Status. Fortunately, I don’t person thing there!
Practical tips & insights
To beryllium capable to marque the astir of your log record analysis, I suggest the following:
- Analyze a longer play of clip to gully meaningful conclusions. I suggest analyzing astatine slightest 1 month’s worthy of data. When analyzing, marque definite to power betwixt antithetic day ranges to spot trends oregon differences connected circumstantial days oregon successful circumstantial weeks.
- Don’t beryllium acrophobic to play with Raw Logs and usage filters to find precisely what you are looking for.
- If your website is not a monolithic one, I promote you to reappraisal each Raw Logs and each Pages successful Logs. It volition beryllium a large workout for you to recognize however bots are crawling your website and what they travel across.
The recap
As you tin see, acknowledgment to awesome tools specified arsenic JetOctopus Log Analyzer, you don’t request to beryllium a super-advanced tech SEO geek to beryllium capable to analyse log files. And you don’t request to bash that successful Excel, either.
If you similar this tutorial, delight stock it with different SEOs oregon stock it connected Twitter.
If you haven’t already, travel maine connected Twitter and subscribe to the SEOSLY YouTube channel. Thank you!
English (US)