About us

I am a Senior Software Engineer at Monotype and write blogs related to software technologies

Comments

Post a Comment

Popular posts from this blog

Rclone: Sync files from ftp server to AWS S3 bucket

Introduction Rclone  is a very good command line tool to move files and directories to and from cloud storage. In this post, I will explain the steps involved in setting up of rclone on mac machine and using it for syncing the files from a free ftp server to AWS S3 bucket Steps Install rclone command line Create source remote(ftp server) Create destination remote(AWS S3 bucket) Use rclone command to sync file from source remote(ftp server) to destination remote(AWS S3 bucket).  Installation of rclone Use curl command to install rclone on mac machine curl https://rclone.org/install.sh | sudo bash Create source remote Here source is ftp server, so a remote needs to be created with ftp configuration. Use command rclone config Follow the instructions and opt FTP when you get option to select storage. Use 'speedtest.tele2.net' as host, 'anonymous' for user and password can be anything. Create destination remote Destination remote is AWS S3 bucket. Fo...
Read more

Amazon Elasticsearch Domain Size Calculation

Introduction In this article, I would be describing the ES(Elasticsearch) terms necessary for understanding the calculation. After acquainting you about Amazon ES, I will tell you about how to calculate the minimum storage size and number of shards required for Elasticsearch with example. Elasticsearch Elasticsearch is an open-source analytics engine for use cases such as log analytics, real-time application monitoring and clickstream analytics. Node A Node is an EC2 instance. Elasticsearch Cluster Elasticsearch Cluster is a collection of one or more node(server) that holds the entire data and provides federated indexing and search capabilities across all nodes. Amazon Elasticsearch Amazon Elasticsearch is a managed service, which makes it easy to deploy, operate and scale Elasticsearch clusters in the AWS cloud. What Amazon Elasticsearch manages Amazon ES provisions all the resources for the Elasticsearch cluster and launches it.  It autom...
Read more

Query S3 bucket using AWS Athena service

Image
Before going into details of how to use Athena service to query S3 bucket, let me introduce you all about what AWS Athena is actually. What is AWS Athena? Athena is query service which uses standard SQL to fetch data from s3 bucket. Use case Suppose there are 3 files stored in s3 bucket (athena-sample-bucket) samplefile1.log.gz(1gb) samplefile2.log.gz(700mb) samplefile3.log.gz600mb Each log.gz file mentioned above has log data in CSV structure with comma-separated fields. Now, what will we do if we want to get data record which belongs to one of the files in s3 bucket? Approach 1 Read each file in s3 bucket and write logic to search the record which we want to fetch. This approach is slow and complex. Approach 2 Use AWS Athena service to query required record from s3 bucket. Athena uses standard SQL. Let's use Athena step by step Goto services section of AWS website and search for Athena. Select Athena service from the search result. This is how i...
Read more