Start your own Amazon Web Service instance for deep learning怎么样开始建一个你自己的亚马逊深度学习机器

I am back to my blogging life after awhile~ 好久没有写博客,我又回来了!


I’ve been working on image classification and segmentation quite a lot recently, and totally in love with GPU big data processing. If you wanna process data that at gigabyte (G) level data definitely look into start a GPU AWS instance 最近我的工作接触了很多图像分类,和图像分割的内容,感觉自己太爱gpu图像分析的世界:太神速了。如果你现在处理的数据已经达到G级别了,我觉得你还是应该开一个亚马逊的ami(亚马逊的深度学习平台/机器)

It is not free, though. You definitely would start with AWS free tier, but I normally use their g or p machines. For example, if I use g2.2 x large, I will be charged about $0.65 per hour.  for more information, go here. It charges by how much you use and if you are new to deep learning and just wanna run some case studies, I think it worths more than building your own GPU machine or buy a new pc with super GPU.

但是话说回来亚马逊的ami其实也不是免费的。我现在用的机器主要两种p和g。比如我现在一般用的是g2.2 x large,价格大概在0.65美金一个小时。更多的选择可以看这里。我觉得这个还是很有吸引力的,如果你只是想要跑几个学习案例的话,我觉得这个ami非常棒。总之还是比现在才在学习阶段,就买台有gpu的电脑或者建自己的gpu机器学习平台有用。


You should definitely do some research on: 在去开个亚马逊深度学习ami之前,我觉得大家该想想:

  1. What do you wanna do with the AWS machine? Do you wanna learn just some basic machine learning stuffs that you only need to process megabyte (?M) level csv/txt data file you could just use your personal computer. A personal computer is fast enough though days. 你想拿这个亚马逊深度学习平台来做什么?如果只是用来处理几兆几十兆的数据的话,那还是没有必要开一个,现在的个人电脑那么快完全可以处理这些数据了。
  2. As I mentioned above, if you wanna process images or data that above some certain level your personal computer could not handle. Think about how much you wanna spend on the data processing. Again, evaluate your situation, needs and do some research. 但是,如果你的数据量已经是在几百兆或者g级别的,当然还是很有必要开一个的。话说回来,还是应该做些调查研究加上考量自己的情况。

My needs for this personal AWS EC2 machine are: 我需要这个亚马逊ami深度学习平台,主要是想用来做:

  1. Processing big data set on neural network image classification and segmentation;图像分类和图像分割;
  2. A machine that has Tensorflow, Theano, Torch, Keras, and also Caffe installed. Tensorflow, Theano, Torch, and Caffe are deep learning ecosystem/environment. Keras is the python module that I use to build deep learning algorithm architecture.想这个ami机器上有我想用的几个深度学习框架,比如Tensorflow, Theano, Torch, and Caffe。还有如果有keras,python的一个构建深度学习/机器学习的包。

If you are thinking about doing the same things, this is a great blog to start your own AWS AMI Instance here or this one. They both have explicit instructions on how to star the instance.


Second options of launching an AWS AMI with a jupyter notebook server without going through all the AWS web console. Using the following command line in your terminal:


Copy and paste the following command lines (CLI) from above figure.

# create security group
aws ec2 create-security-group –group-name JupyterSecurityGroup –description “My Jupyter security group”

# add security group rules
aws ec2 authorize-security-group-ingress –group-name JupyterSecurityGroup –protocol tcp –port 8888 —cidr
aws ec2 authorize-security-group-ingress –group-name JupyterSecurityGroup –protocol tcp –port 22 —cidr
aws ec2 authorize-security-group-ingress –group-name JupyterSecurityGroup –protocol tcp –port 443 —cidr

# launch instance
aws ec2 run-instances –image-id ami-41570b32 –count 1 –instance-type p2.xlarge –key-name <YOUR_KEY_NAME> –security-groups JupyterSecurityGroup

The next thing would be to configure your Jupyter Notebook Server:


jupyter notebook –generate-config
key=$(python -c “from notebook.auth import passwd; print(passwd())”)

cd ~
mkdir certs
cd certs
openssl req -x509 -nodes -days 365 –newkey rsa:1024 –keyout mycert.key -out mycert.pem

cd ~
sed -i “1 a\
c = get_config()\\
c.NotebookApp.certfile = u’$certdir/mycert.pem’\\
c.NotebookApp.keyfile = u’$certdir/mycert.key’\\
c.NotebookApp.ip = ‘*’\\
c.NotebookApp.open_browser = False\\
c.NotebookApp.password = u’$key’\\
c.NotebookApp.port = 8888″ .jupyter/

These CLI are to create your AWS AMI certificate for Jupyter Notebook server, and then you could run and test out if your jupyter notebook works, after seccessfully run above CLI.

screen -S jupyter
mkdir notebook
cd notebook
jupyter notebook

For more info you could see this blog for details.




A baby step towards my interactive map application using Leaflet JavaScript


This is how my national GDP interactive map looks like on the local host. You could watch my first ever video record on YouTube of this interactive map. A brief introduction of the map and also the codes using HTML, CSS, and JavaScript. This was a very simple example I made to test some of my ideas.

If you remember my last blog that I present an interactive map host via ESRI ArcGIS Online.  After my data was successfully uploaded, I found several issues that I don’t like about it:

  1. Even though ESRI ArcGIS Online have a super nice format that you could visualize the spatial data in a pretty way, but the data loading from the site is very slow, AND IT’S COULD BE VERY EXPENSIVE. I am at my 60 days free trial at this point and I believe if I wanna use the server and do some data analysis on ArcGIS Online I have to buy their credits;
  2. The way of data presenting is restricted to the certain format depends on how you select the web map format from ESRI.

I use quite a bit of R, and I know that there are two packages in R called Shiny and Leaflet For R might help me develop the idea. I was so thrilled to find these packages, I feel a bright light shine on my road and point to the destination I wanna head to, and I found a perfect example that my web map application will look like especially the case of  American Super Zipcode. There are not only an interactive map but also while you zoom in and out you could also show some statistic results on the right side of the map. It’s too cool.

But I was so disappointed too while I found out developing a web application through Shiny and Leaflet for R would not be free, because I still need a server to host my data and APP once they could be share. However, at the point that I only need to test my ideas.

I gave up the two methods I found above and even checked out Mapbox Studio and Cartodb, two of the most popular online interactive map and visualization platform. But they are for developers (you still could use it without coding background, though), but I wanna have some features that require coding in Javascript. Leaflet JavaScript library is the last and best way I could use, which could give me enough freedom to figure out the functions/features for my application, and even the interactive analytical tools that I could put up over there. Now I also find D3 might be even more attractive because it hosts a bigger JS library that not only for the interactive map but also other online interactive way of data visualization.

I got a lot help from briefing through some YouTube videos (that’s the reason I recorded a video myself and hope it could be helpful to another struggling beginner like myself). Learn quite a lot of new things like GeoJson and GeoJson-vt. GeoJson is a geodata format for JavaScript, which is equal to shapefile for ArcGIS and QGIS. If your dataset is bigger than 1 M, the data loading to your website would slow down, so the founder of leaflet JS library wrote a vector tile JS codes (GeoJson-vt) to speed up the shapefile data loading process.

Here is my HTML, CSS and JavaScript code for the application you see in the video, You could also find me and my codes on GitHub


Esri technical certification: preparation tips

I find this is super helpful!!!

Alex Tereshenkov

Over last two years I have passed a bunch of Esri technical certification exams (10 exams to be precise). Esri did a great job posting multiple blog posts regarding how to prepare for the certification: there is a free training seminar covering the details and some blog posts and Esri Australia blog post. Esri has also authored two special instructor-led courses for those who plan to become certified as ArcGIS Desktop Associate and ArcGIS Desktop Professional:
Esri Technical Certification: Skills Review for ArcGIS Desktop Professional
Esri Technical Certification: Skills Review for ArcGIS Desktop Associate

Another useful resources are free web courses (2 for Deskop and 1 for Server):
Esri Technical Certification: Sample Questions for ArcGIS Desktop Associate

Esri Technical Certification: Sample Questions for ArcGIS Desktop Professional

Esri Technical Certification: Sample Questions for Enterprise Administration Associate

I highly recommend going through sample questions to get an idea of what…

View original post 840 more words