To Nha Notes | Sept. 14, 2023, 8:13 p.m.
In AL2023 the logging system package has changed from Amazon Linux 2. AL2023 doesn't install rsyslog by default, so the text based log files such as /var/log/messages that were available in Amazon Linux 2 aren't available by default. The default configuration for AL2023 is systemd-journal, which can be examined using journalctl. Although rsyslog is an optional package in AL2023, we recommend the new systemd based journalctl interface and related packages. For more information, see the journalctl manual page.
journalctl is a command-line utility in Amazon Linux 2023 that allows you to query and display log messages from the systemd journal. The systemd journal is a centralized logging system that collects and stores log data from various sources, including system services, kernel events, and user applications. With journalctl, you can read logs, monitor the logs in real-time, filter the logs based on time, service, severity, and other parameters.
To use journalctl, open a terminal window and type journalctl followed by any options you want to use. Press Enter to execute the command. Here are some examples of how to use journalctl:
To forward journalctl logs to AWS CloudWatch Logs, you can use the following steps:
[/var/log/journal]
log_group_name = my_log_group
log_stream_name = {instance_id}
file = /var/log/journal/%m-%d-%y/system.journal
datetime_format = %b %d %H:%M:%S
https://github.com/saymedia/journald-cloudwatch-logs
https://gist.github.com/RichardHightower/318971f60e2dd44ef71027ce5ea2cb05
https://advantageous.github.io/systemd-cloud-watch/
https://github.com/advantageous/systemd-cloud-watch
http://rick-hightower.blogspot.com/2017/03/systemd-cloud-watch-to-send-linux-logs.html
mkdir /tmp/logagent cd /tmp/logagent curl -OL https://github.com/saymedia/journald-cloudwatch-logs/releases/download/v0.0.1/journald-cloudwatch-logs-linux.zip unzip journald-cloudwatch-logs-linux.zip sudo mv journald-cloudwatch-logs/journald-cloudwatch-logs /usr/bin sudo mkdir -p /var/lib/journald-cloudwatch-logs/ sudo chown -R ec2-user /var/lib/journald-cloudwatch-logs/
/etc/journald-cloudwatch.conf (root: 644)
cat << 'EOS' >> /etc/journald-cloudwatch.conf
aws_region = "${instance.Region}"
ec2_instance_id = "${instance.InstanceID}"
log_group = "project-loggroup"
log_stream = "project-logstream"
state_file = "/var/lib/journald-cloudwatch-logs/state"
log_priority = "DEBUG"
EOS
/etc/systemd/system/journald-cloudwatch.service (root: 664)
cat << 'EOS' >> /etc/systemd/system/journald-cloudwatch.service
[Unit]
Description=journald-cloudwatch-logs
Wants=basic.target network-online.target
After=basic.target network-online.target
AssertPathExists=/var/lib/journald-cloudwatch-logs
[Service]
User=ec2-user
Group=ec2-user
ExecStart=/usr/bin/journald-cloudwatch-logs /etc/journald-cloudwatch.conf --debug start
KillMode=process
Restart=on-failure
RestartSec=10s
[Install]
WantedBy=default.target
EOS
systemctl enable journald-cloudwatch.service
systemctl start journald-cloudwatch.service
systemctl status journald-cloudwatch.service
journalctl -xe
https://github.com/saymedia/journald-cloudwatch-logs
https://gist.github.com/RichardHightower/318971f60e2dd44ef71027ce5ea2cb05
{
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
"log_group_name": "amazon-cloudwatch-agent.log",
"log_stream_name": "amazon-cloudwatch-agent.log",
"timezone": "UTC"
},
{
"file_path": "/opt/aws/amazon-cloudwatch-agent/logs/test.log",
"log_group_name": "test.log",
"log_stream_name": "test.log",
"timezone": "Local"
}
]
}
},
"log_stream_name": "my_log_stream_name",
"force_flush_interval": 15
}
}
https://linuxhandbook.com/journalctl-command/
https://betterstack.com/community/guides/logging/how-to-control-journald-with-journalctl/
https://www.geeksforgeeks.org/journalctl-command-in-linux-with-examples/
https://linuxconfig.org/journalctl-command-usage-and-examples-on-linux
https://www.freedesktop.org/software/systemd/man/journalctl.html
https://github.com/saymedia/journald-cloudwatch-logs
https://crates.io/crates/journald-to-cloudwatch
https://github.com/nicholasbishop/journald-to-cloudwatch
https://devopscube.com/how-to-setup-and-push-serverapplication-logs-to-aws-cloudwatch/
https://www.virtualizationhowto.com/2023/09/best-open-source-log-management-tools-in-2023/
https://blog.swwomm.com/2021/06/send-journald-to-cloudwatch-logs-with.html