Tag: aws
When using AWS, what tools and services are essential for my app?
Amazon Web Services (AWS) is a cloud-based computing platform owned by Amazon.com. It provides services and tools for software developers that allow them to concentrate on coding without needing to worry about the required infrastructure that must be in place for them to do so. The types of services that AWS provides include compute services, storage services, database services, networking services, management services, analytics services, security services, and application services. The tools and services that AWS offers are vast, with over 200 currently to choose from. If you choose to use the AWS platform to develop your app, however, there are a few specific services that are utilized by just about every app that is developed. They include the following:
- Amazon EC2 (Amazon Elastic Compute Cloud)
- Amazon EBS (Elastic Block Storage)
- Amazon S3 (Amazon Simple Storage Service)
- Amazon RDS (Amazon Relational Database Service)
In this article, we will give an explanation of each service as well as what the potential cost of using the service might be.
Amazon Elastic Compute Cloud (EC2)
Amazon EC2 provides secure, resizable compute capacity in the cloud which basically amounts to users renting virtual machines. Each virtual machine, which is called an “instance,” is then loaded with the operating system of the user’s choice and can be used to run applications, to store and process data, and to host websites. Amazon EC2 can also be used to launch apps quickly and efficiently and to manage them across multiple servers.
Amazon EC2 provides a wide selection of instance types which include the following:
- Amazon EC2 A1 Instances. Powered by AWS’s custom-designed Graviton2 processors. They are optimized for scale-out workloads such as containerized microservices, web servers, gaming, and media processing.
- Amazon EC2 C5 Instances. Optimized for compute-heavy workloads. They are the ones to consider for high-performance computing, machine learning, and video encoding.
- Amazon EC2 M5 Instances: Designed for general purpose workloads such as web servicers, batch processing, and gaming.
- Amazon EC2 R5 Instances. Optimized for memory-intensive workloads such as in-memory database, distributed web scale in-memory caches, and real-time big data analytics.
- Amazon EC2 T3 Instances. Designed for burstable workloads such as web servers, development environments, and small databases.
For these instances, the user can choose from varying combinations of CPU, memory, storage, and networking capacity. This allows the user to pick the mix of resources that will best meet their needs. It also provides users with a range of security and networking options which allows them to configure their instances to meet their specific security and networking requirements. Amazon EC2 is designed to help businesses scale and grow by providing access to as much power as they need on demand and by charging for only the memory that is used. It also completely eliminates the need for expensive hardware that can take long amounts of time to set up and configure.
AWS offers a free tier of EC2 Linux and Windows micro instances for the first 12 months of service which includes 750 hours of service per month. EC2’s prices vary depending on the instances chosen and the time you spend using the instance. They charge by the hour or the second depending on the instance you run. There are no upfront payments or commitments required. To save money, AWS also offers spot instances. Spot instances use spare Amazon EC2 computing capacity at a huge discount. You can also sign up for their savings plan, which will provide a discount for using a consistent amount of usage over an extended period of time. To qualify for the savings plan, you do have to sign a commitment.
Amazon Elastic Block Storage (EBS)
Amazon EBS provides block-level storage for the data associated with the applications that are run by Amazon EC2. It is basically a virtual hard drive. Amazon EBS stores data that needs to be accessed quickly and frequently. This data can include databases, log files, application data, and media files and is persistent. This means that the data will remain there even if the computer or server it is being served from is powered down.
Amazon EBS tends to be more reliable than traditional hard drives and has a number of other advantages:
- It can also be accessed from anywhere since the information is stored on the cloud.
- It can be resized without taking the instance that is being used offline.
- It can be backed up and restored with just a few clicks.
- The data on an EBS system can be moved from one EC2 instance to another.
In order for Amazon EBS to be used with an instance, a user must determine the volume or capacity that is needed and attach it to an instance in Amazon EC2.
AWS offers a free tier of EBS which includes 30 GB of storage per month in addition to 2 million input/output operations per second (IOPS) and 1 GB of snapshot storage. Snapshot storage are point-in-time copies of your block data. If you go over the 30GB of free storage, the standard charge is $0.05/GB-month. EBS also offers different types of storage, all of which come with their own price.
Amazon Simple Storage System (S3)
Amazon S3 is a cloud storage platform that provides an easy way to store and retrieve files from the cloud. It can be accessed anywhere there is an internet connection and is ideal for storing large amounts of data such as photos, videos, music, and documents among other files. It is also cost effective because the user only pays on the actual amount of data being stored.
Data stored using S3 are placed into large “folders” called buckets. To use S3, you create a bucket, assign it a name, set its permissions and access controls, and then assign data to it. The user can control who has access to their data and set up automatic backups. The items placed in the bucket are encrypted, highly secure, and are available via the internet using HTTP.
When using S3, you pay to store items in your buckets, and the rate you’re charged has many variables to consider. Those variables include the size of the items you’re storing, for how long you are storing them. S3 also has different storage classes which are based on how often you need to access your data, and the rate varies depending on the class or classes you choose. AWS offers a pricing calculator that will help you get an estimate of the AWS services you intend to use.
Amazon Relational Database Service (RDS)
Amazon RDS is a cloud-based service that provides users with a relational database for their applications. In a relational database, data is organized into tables with columns and rows. The columns represent attributes of data and the rows contain records or instances with those attributes. This allows users to quickly and easily access data from multiple sources without needing to search through multiple tables or needing to install and manage a database. Amazon RDS also allows users to easily replicate and make backups of their data.
Amazon charges for RDS services based on the total data that is transferred from all sources for backups, replication, and access over the Internet. The price is also dependent on the instance types you have, your database engine, and the regions in which the data is stored. Using RDS can sometimes become a bit somewhat pricey. If you are on a budget or just wish to avoid the cost, you can look into using MySQL instead and determine whether it may suit your needs. MySQL is open-source and free, but it requires users to have a good understanding of database concepts and administration. MySQL will not always be a suitable alternative, however, and RDS does charge to run MySQL. Plus, switching from one relational database service to the other can be difficult, costly, and result in lost data, so you may want to wisely choose which service to use. It is often best to choose which one you believe will suit your needs and then stick with it.
Matraex is a premier app and software development company located in Boise, Idaho. We want to help you become informed in app development concepts so you can make informed decisions and be a consumer who is “in the know”. Feel free to leave us a question on our Google Business Profile, contact us with your questions, or leave a message in the chat feature on our website. We look forward to answering your questions.
Sign up to receive answers to your questions delivered directly to your inbox!
What Is Amazon Web Services (AWS)?
Amazon Web Services, also known as AWS, is a cloud-based computing platform that is owned and operated by Amazon.com. It offers a wide range of tools and services that can help businesses become more agile and increase their innovation.
AWS Services include computation, storage, analytics, networking, and mobile services. AWS is the world’s most popular cloud computing platform with over a million customers in 190 countries. It is used by businesses, from startups to large corporations, organizations, and institutions to build applications.
AWS allows companies to quickly and easily build, manage, and scale applications and services. It can be accessed from anywhere that has internet access. AWS does this by providing needed infrastructure so developers can focus on their projects.
With AWS, developers do not need to worry about storing and analyzing data, how content is going to be delivered, or where they are going to create code. AWS provides flexible and cost effective solutions to these types of problems. This helps companies reduce cost while increasing efficiency.
Why is AWS useful to app developers?
Amazon Web Services is an important tool for developers. It provides a cloud platform that allows them to quickly create reliable and scalable applications. With its wide range of storage, database, and networking services, it allows developers to write code for their applications without worrying about the underlying infrastructure.
It also provides cost-effective, pay as you go pricing. This makes it attractive for developers who want to build applications quickly and cost effectively. It also allows them to scale their applications at a later date.
Why do developers request administrative access to AWS?
Developers request administrative access to Amazon Web Services to facilitate faster development. While the developers are setting up servers, there is often a complex relationship between the numerous resources and services that need to be established and maintained. Allowing administrative access enables developers to set up those relationships between the different systems without needing to communicate why each system is required.
In addition, it may be difficult for an app owner or manager to understand the technicalities involved with AWS and the relationships that needs to be established between different services. It may be simpler, faster, and less confusing to give the developer the access they need. AWS offers foundational services including compute power, network, storage, gateway load balancing, and data warehouse services. AWS offers over 200 total services, which can be overwhelming and confusing.
Administrative access to AWS is also requested so developers can manage advanced security systems. They may also need administrative access so they can monitor and troubleshoot issues when they arise. Without access, they would need to request individual permissions every time something new comes up.
Do developers require administrative access to AWS?
App and software developers may not require administrative access to AWS, but without it, the product owner or product manager may need the knowledge to manipulate some of the AWS services on their own. Without administrative access, the product owner or manager may also need to give multiple permissions for individual access to various AWS tools and services as their app is being developed.
If the developer cannot access to the tools and services they need to implement and integrate into the app and does not give the developer permission, the product owner or manager would need to do it themselves.
What are the most commonly used tools and services of AWS?
The most common features of Amazon Web Services include the following:
- Amazon Elastic Compute Cloud (EC2): A web service that allows users to easily launch and manage virtual machines, or instances, in the cloud. With EC2, users can quickly create a virtual machine and load the machine with their own software. The machine can then be scaled up or down as needed.
- Amazon Elastic Block Storage (EBS): A service that provides secure, high performance storage for Amazon Elastic Compute Cloud (EC2) instances. EBS provides persistent block-level storage volumes that can be attached to and detached from EC2 instances as needed, allowing for dynamic storage to meet application needs.
- Amazon Simple Storage Service (S3): A highly reliable, cost effective storage service that can be used to store data on the cloud. S3 provides a simple web interface for users to store and retrieve any amount of data from anywhere on the web.
- Amazon Relational Database Services (RDS): A managed cloud data service that allows users to quickly launch and manage relational databases in the cloud. Relational databases are organized into tables with rows and columns, and each item on the table is related to one another. RDS is highly scalable and provides many features that help users manage their databases more efficiently.
- Amazon Elastic Load Balancing (ELB): A web service that helps distribute traffic in an application. It automatically routes traffic to multiple instances of the application so they are not all bogged down on one. This provides fault tolerance because it helps to ensure the application is always available to users and can handle incoming requests.
- Amazon CloudFront: A content delivery network(CDN) that helps to deliver content to users around the world. It provides low latency and high transfer speeds.
- Amazon DynamoDB: A NoSQL database service that is highly available, scalable, and secure. A NoSQL database is a non-relational database that stores data in an unstructured way. DynamoDB provides high performance and low latency for applications that need to access large amounts of data.
- Amazon Lambda: A serverless computing service that allows users to run code for applications without needing to manage underlying infrastructure. Lambda is perfect for applications that need to scale quickly and can handle unpredictable workloads.
- Amazon Machine Learning (ML): A service that allows users to easily build, train, and deploy machine learning models in the cloud. This helps to reduce the time and cost associated with building and managing custom machine learning solutions.
- Amazon Redshift: A cloud data warehouse that makes it easy to store and analyze large amounts of data. It is highly scalable and provides a fast, cost-effective way to analyze data.
- Amazon Simple Queue Service (SQS): A message queuing service that helps to manage message-based communication between applications. SQS helps to ensure that messages are delivered in the correct order. It also allows for asynchronous communication between the different components of an application.
- Amazon Simple Notification Service (SNS): A highly scalable push notification service that is easy to set up and operate. It is highly scalable and is a simple and efficient way to fan-out messages to subscribers. SNS allows for notifications to be sent to varying number of recipients with no additional set-up required. It can provide notifications through mobile devices, email, and social media.
Go to “When using AWS, what tools and services are essential for my app?” for specific detail on these and other services.
How does AWS charge for their services?
Matraex is an app development company in Boise, Idaho. Do you have any app development or tech questions? We’d like to be your source for unbiased answers so you be an informed consumer.
Contact Matraex, call us at (208) 344-1115, direct message a question on our website, or put a question on our Google Business Profile. We look forward to talking with you.
Sign up to receive answers to your questions delivered directly to your inbox!
AWS Auto Scaling Group – CodeDeploy Challenges
AWS Auto Scaling Group – CodeDeploy Challenges
First here is my setup
- A single development / test server in the AWS cloud, backed by a separate Git Repository.
- WHen code is completed in the development environment it is commited to the development branch (using whichever branching scheme best fits the project)
- At the same time the code is merged to the test branch, and the code is available for client testing on the ‘test_stage’ site if they would like
- Then on an as needed basis the code in in the test branch (on the test_stage server) is deployed to AWS using their CodeDeploy api
- git archive test -> deploy.zip
- upload the file an S3 bucket (s3cmd)
- register the zip file as a revision using the AWS Register Revision API call
- This creates a file that can be deployed to any deployment group
- I setup two groups in my AWS account , test and live.
- When the client is ready, I run a script which deploys thes the ziped up revision to the Test server, where they are able to look atit and approve.
- Then I use the same method but move it instead of the www deployment group.
(The complexities of setting this up are deeper than I am going in this article, but for future prospects, all of this programming knowledges is stored in our deploy.php file)
A couple of tricks “they” dont tell you.
- Errors can be difficult to debug – if you update your code deployment to do more verbose logging it can help you to determine what some of the errors were.
- update /etc/codedeploy-agent/conf/codedeployment.yml, set verbose to yes.
- restart the service /etc/init.d/code-deployment restart (it can take several minutes to restart, this is normal)
- tail the log files to watch a deployment in real time, or investigate it after the fact (tail /var/log/aws/codedeploy-agent)
- Deploying a Revision to servers while they may be going through some termination instability, may likely cause your deployment to fail when one of you servers terminates.
- To prevent this, update the deployment autoscaling plan to have a minim and a maximum of the server, and do not take it under load during the 10 – 15 minutes (up to 2 hours) issues will cause errors
- Depending on the load on your servers, your deployment could take a lot of cpu and could generate an autoscaling alert and could spin up new tasks or send you an email. There is not a correct way to deal with this, however it is a good idea to know about it before you deploy.
- Finally the item that I wrote this because of, it appears that when you attempt to deploy a revision to an autoscaling group, it can cause some failures.
- The obvious one is that the deployment will fail if it is attempted while the server is shutting down
- However, it seems that if you have decided to upgrade your AMI, and your Launch Configuration, that a deployment will fail. And for me, it actually caused a key failure to login as well (this could have been because of multiple server terminations and then another server took over the IPs within a few minutes) Anyway, much caution about these things.
UPDATE:
Well, the problem was actually that the by ‘afterinstall.sh’ script, was cleaning up the /opt/codedeployment/ directory (so we didn’t run out of space after a couple dozen deployments), but I was also removing the appspec.yml file.
So I updated the command that runs in the afterinstall to be
/usr/bin/find /opt/codedeploy-agent/deployment-root/ -mindepth 2 -mtime +1 -not -path '*deployment-instruction*' -delete
Debugging CodeDeployment on AWS
Debugging CodeDeployment on AWS
This article is being written well after I have already installed the CodeDeployment daemon on an ubuntu server, created an AMI out of it and set it up as an auto launch server from a Scaling Group.
I am documenting the process I went through to dig into the error a bit more, this helps to identify and remember where the logs files are and how to get additional information, even if the issue is never the same again.
Issues with running the codedeployment showed up as a python error in the log
more /var/log/aws/codedeploy-agent/codedeployment-agent.log
put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{"error_code":5,"script_name":"","message":"not opened for reading","log":""}"
I decided this is not enough information to troubleshoot an error so I had to dig in and fina way to make it more verbose. I found this file, just update verbose from false to true
vi /etc/codedeploy-agent/conf/codedeployagent.yml
Then restart the codedeploy-agent
/etc/init.d/code-deployagent restart
This can take quite a while since it runs quite a bit of background installing and checking for duplicate processes. but once it is complete you can check that the process is running again.
ps ax|grep codedeploy
Once this is running in verbose mode, monitor the log
tail -f /var/log/aws/codedeploy-agent/codedeployment-agent.log
and re-run the deployment and view the results of the log, the most useful thing for me was to grep for the folder that more specific error information was written to.
grep -i "Creating deployment" /var/log/aws/codedeploy-agent/codedeployment-agent.log
This showed me the folder that all of the code WAS going to be extracted to, since there was an error the system actually dumped the contents of an error into a file called bundle.tar in the folder that it would have exported to.
cat /opt/codedeploy-agent/deployment-root/7ddce865-0611-45f0-bf74-459fcf806f23/d-YK4NWBJD7/bundle.tar
This returned an error from the S3 showing that the Code Deploy was having an error downloading from S3, so I had to add access to the policy to download from S3 buckets as well