Server Time and Clock Sync: Small Mistakes, Big Consequences

Most ready made operating system images ship set to UTC, while Saudi Arabia runs on UTC+3 all year with no daylight saving. Miss that difference on a new server and every cron job you scheduled for 2:00 AM actually runs at 5:00 AM Riyadh time, NOW() in your database returns a time 3 hours behind the wall clock, and timestamps in emails and logs mislead everyone who reads them. The second problem is quieter and more dangerous: clock drift. An unsynchronised server clock can drift several seconds a day, which breaks two factor codes, gets TLS certificates rejected and makes tracing an incident across several servers' logs close to impossible. The fix has three steps. Set the system zone with timedatectl set-timezone Asia/Riyadh if your crons and tools assume Riyadh time, enable synchronisation through chrony so drift stays within fractions of a millisecond, then confirm that the system, the database and the application agree. The golden rule for applications is to store time in UTC in the database and convert it to Riyadh time only when you display it. This article gives the full commands for each step, a quick check table and the mistakes we see most often.

What Actually Happens When a Server Arrives on UTC

The scenario repeats itself often. A team moves its system from an old server that was set to Riyadh time onto a new one, and everything looks fine on day one. A few days later the symptoms appear:

• Crons run 3 hours late. The sales report scheduled for 7:00 AM arrives at 10:00 after the workday has started, and the backup scheduled for 1:00 AM starts at 4:00 and collides with the first morning traffic.
• The database writes a different time. Every row using NOW() or CURRENT_TIMESTAMP is stored 3 hours apart from the migrated rows, and chronological ordering in reports gets scrambled.
• Emails carry misleading timestamps. Some systems print the server clock into the message body with no zone, so a customer reads that the invoice was issued at 6:00 when it was issued at 9:00.
• The day ends in the wrong place. A "today's sales" report counting from midnight drops the last 3 hours of the real day and adds the first 3 hours of the next one.

Each symptom looks like a small, separate bug, yet they share one cause, and diagnosing it starts with a single command.

Check the Current State in 30 Seconds

timedatectl
date
date -u

timedatectl tells you three essential things: Time zone, the current zone; System clock synchronized, whether the clock is actually in sync; and NTP service, whether sync is active. If you see Etc/UTC (UTC, +0000) while your team writes crons in Riyadh time, you have found the root cause. To change the zone:

timedatectl set-timezone Asia/Riyadh
systemctl restart cron    # crond on the RHEL family

Restarting cron matters because some versions only read the zone at startup. If you want to keep the system on UTC and run specific jobs on Riyadh time, cronie supports a CRON_TZ=Asia/Riyadh line at the top of the crontab, although support varies between distributions, so test it first.

Syncing With chrony: Why the Clock Alone Is Not Enough

Every machine's clock relies on a crystal oscillator that never matches the ideal frequency exactly. A drift of 50 parts per million, which is normal, means about 4.3 seconds per day, or more than two minutes over 30 days. Virtual machines are even more exposed because the clock can jump after a migration or a pause.

That drift breaks real things:

System	Rough tolerance	What happens beyond it
TOTP two factor codes	30 second window	Valid codes rejected, login complaints
Kerberos and Active Directory	5 minutes	Authentication fails completely
TLS certificates	Based on start and expiry dates	HTTPS refused for a certificate "not yet valid"
Database replication	Seconds	Misleading lag readings and wrong event order
Log analysis across servers	Under a second	An incident timeline that looks physically impossible

chrony is the default on modern distributions. To install and enable it:

# Ubuntu and Debian
apt install chrony
# AlmaLinux and Rocky
dnf install chrony

systemctl enable --now chronyd   # the service is named chrony on Ubuntu

The configuration lives in /etc/chrony.conf or /etc/chrony/chrony.conf, and these are the lines that matter:

pool pool.ntp.org iburst maxsources 4
server time.google.com iburst
makestep 1.0 3
rtcsync

iburst cuts the first sync after boot down to seconds. makestep 1.0 3 lets the clock jump directly if it is off by more than one second during the first 3 updates, after which chrony corrects gradually by speeding up or slowing down the clock so time never runs backwards during operation, which matters because backwards time confuses databases and logs. rtcsync keeps the hardware clock updated so the server boots with a nearly correct time.

To verify:

chronyc tracking
chronyc sources -v

In tracking, look at the System time line, which should show an offset in fractions of a millisecond, such as 0.000213 seconds. In sources at least one source should carry the ^* marker, meaning it is the source currently selected.

System, Database and Application: Three Clocks That Must Agree

Setting the system alone is not enough, because the database and the application each have their own zone setting. In MySQL and MariaDB:

SELECT @@global.time_zone, @@session.time_zone, @@system_time_zone,
       NOW(), UTC_TIMESTAMP();

If time_zone is SYSTEM, the database follows the OS, and changing the system zone changes NOW() after the database service restarts. To pin it explicitly, add default-time-zone = '+03:00' to the config, or use the name Asia/Riyadh after loading zone tables with mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql.

Watch the difference between MySQL column types. A TIMESTAMP column is stored internally as UTC and converted using the session zone on read and write, while DATETIME is stored literally with no conversion. That is why displayed TIMESTAMP values can shift suddenly after a zone change while DATETIME values stay put, even when they were written in another zone.

PHP has date.timezone in php.ini, and Laravel has timezone in config/app.php. Node.js follows the TZ environment variable or the system zone. One quick check covers all three layers:

date '+%F %T %Z'
php -r 'echo date("Y-m-d H:i:s T"), PHP_EOL;'
mysql -e "SELECT NOW(), @@global.time_zone;"

If the three times differ by anything you did not intend, you are looking at a fault that will show up in reports sooner or later.

The Golden Rule: Store in UTC, Display in Riyadh Time

Setting everything to Riyadh time and moving on can look simpler, and for a small internal system that will never leave Saudi Arabia it is acceptable. For any serious system we recommend this approach:

1. Database and application on UTC. In Laravel, for example, keep 'timezone' => 'UTC', so every stored time is unambiguous.
2. Convert at display time only. When showing a time to a user, convert it in one line such as $order->created_at->timezone('Asia/Riyadh').
3. Write crons deliberately. If the system stays on UTC, schedule a job at 23:00 to run at midnight Riyadh time, and document that in a comment above the line.

Why bother? UTC never changes and has no daylight saving, so any integration with an external party, a branch in another country or a cloud service running on UTC lines up directly with no hidden conversions. Even more important, moving the database to another server never changes the meaning of a single row.

A Checklist for Every New Server

1. Run timedatectl and decide explicitly between UTC and Asia/Riyadh, then write the decision into the server's documentation.
2. Enable chrony and confirm a source marked ^* with an offset under 10 milliseconds.
3. Check NOW() and UTC_TIMESTAMP() in the database and confirm the gap is zero or 3 hours, matching your decision.
4. Check the PHP or Node.js zone and make sure it matches.
5. Review every crontab line and ask which zone it was written for.
6. After the first 24 hours, compare actual cron run times in the logs with what you expected.

When you launch a CLOUD cloud server in Riyadh with around 8ms latency, this checklist starts in the first minute and takes under 10 minutes, saving you weeks of confusing reports. The entry package starts at 3,450 SAR per year including VAT, and every package is on the pricing page.