2022年8月15日月曜日

Automatically set timezone when you use Dockerfile

 In the other day , I tried to build a docker image using following Dockerfile.

But I was prevented from a cause I didn't know.


Dockerfile I used
FROM ubuntu:latest

RUN apt-get update \ 
&& apt-get install -y clang clang-12 \ 
&& apt-get install -y neovim git wget curl imagemagick wireshark tshark unzip gdb net-tools

I show a measure for it.

 

What I did


I modified the Dockerfiles as follows , then I was successful in building a docker images.
 
More precisely, I added the line startting from 'ENV' , installing tzdata before installing some package.
 
Modified Dockerfile  
FROM ubuntu:latest

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update \ 
&& apt-get install -y tzdata \
&& apt-get install -y clang clang-12 \ 
&& apt-get install -y neovim git wget curl imagemagick wireshark tshark unzip gdb net-tools

2022年8月12日金曜日

Several errors I encountered when I constructed Spring boot with Docker

 I show several errors I encountered when I constructed Spring boot with Docker and its cause.


You'd better read this article refering my previous article How to run SpringBoot as docker container

Build Error

Unable to find a single main class from the following candidates [hello.Application, hello.Application2] -> [Help 1]


Quotation of part of error message
[INFO] --- spring-boot-maven-plugin:2.7.1:repackage (repackage) @ spring-boot-docker-complete ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15.785 s
[INFO] Finished at: 2022-08-12T10:06:59Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.7.1:repackage (repackage) on project spring-boot-docker-complete: Execution repackage of goal org.springframework.boot:spring-boot-maven-plugin:2.7.1:repackage failed: Unable to find a single main class from the following candidates [hello.Application, hello.Application2] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
As above error message says that "Unable to find a single main class from the following candidates [hello.Application, hello.Application2] " , this cause is that there two main method in a SpringBoot project.


More precisely,

Application.java
@SpringBootApplication
@RestController
public class Application {

        @RequestMapping("/")
        public String home() {
                return "Hello Docker World";
        }

        public static void main(String[] args) {
                SpringApplication.run(Application.class, args);
        }
Application2.java
 @SpringBootApplication
@RestController
public class Application2 {

        @RequestMapping("/test")
        public String home() {
                return "Hello Docker World";
        }

        public static void main(String[] args) {
                SpringApplication.run(Application.class, args);
        }

}

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile (default-compile) on project spring-boot-docker-complete: Compilation failure: Compilation failure: [ERROR] /srv/gs-spring-boot-docker/complete/src/main/java/hello/TestDB.java:[10,37] package org.springframework.jdbc.core does not exist

Quotation of part of error message
 [INFO] --- maven-compiler-plugin:3.10.1:compile (default-compile) @ spring-boot-docker-complete ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 3 source files to /srv/gs-spring-boot-docker/complete/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /srv/gs-spring-boot-docker/complete/src/main/java/hello/TestDB.java:[10,37] package org.springframework.jdbc.core does not exist
[ERROR] /srv/gs-spring-boot-docker/complete/src/main/java/hello/TestDB.java:[22,9] cannot find symbol
  symbol:   class JdbcTemplate
  location: class hello.TestDB
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  7.272 s
[INFO] Finished at: 2022-08-12T11:03:58Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile (default-compile) on project spring-boot-docker-complete: Compilation failure: Compilation failure: 
[ERROR] /srv/gs-spring-boot-docker/complete/src/main/java/hello/TestDB.java:[10,37] package org.springframework.jdbc.core does not exist
[ERROR] /srv/gs-spring-boot-docker/complete/src/main/java/hello/TestDB.java:[22,9] cannot find symbol
[ERROR]   symbol:   class JdbcTemplate
[ERROR]   location: class hello.TestDB
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

As above error message says that "package org.springframework.jdbc.core does not exist" or "cannot find symbol symbol: class JdbcTemplate location: class hello.TestDB", this cause is that although TestDB tries to use JDBC API , JDBC library doesn't exist in the SpringBoot project. More precisely, pom.xml is as follows.
 ### we can not fine description about JDBC !!
### Omit
        <dependencies>
                <dependency>
                        <groupId>org.springframework.boot</groupId>
                        <artifactId>spring-boot-starter-web</artifactId>
                </dependency>

                <dependency>
                        <groupId>org.springframework.boot</groupId>
                        <artifactId>spring-boot-starter-test</artifactId>
                        <scope>test</scope>
                                                                </dependency>

                <dependency>
                       <groupId>mysql</groupId>
                       <artifactId>mysql-connector-java</artifactId>
                       <scope>runtime</scope>
                </dependency>


        </dependencies>
        
### Omit        

Runtime Error

Invocation of init method failed; nested exception is java.lang.IllegalStateException: Ambiguous mapping. Cannot map 'application2' method

Quotation of part of error message
 2022-08-12 11:25:46.849  INFO 870 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2022-08-12 11:25:46.913  INFO 870 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2022-08-12 11:25:46.917  INFO 870 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.64]
2022-08-12 11:25:47.210  INFO 870 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2022-08-12 11:25:47.211  INFO 870 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 6225 ms
2022-08-12 11:25:48.584  WARN 870 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'requestMappingHandlerMapping' defined in class path resource [org/springframework/boot/autoconfigure/web/servlet/WebMvcAutoConfiguration$EnableWebMvcConfiguration.class]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Ambiguous mapping. Cannot map 'application2' method 
hello.Application2#home()
to { [/]}: There is already 'application' bean method
hello.Application#home() mapped.
2022-08-12 11:25:48.601  INFO 870 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2022-08-12 11:25:48.658  INFO 870 --- [           main] ConditionEvaluationReportLoggingListener : 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2022-08-12 11:25:48.767 ERROR 870 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'requestMappingHandlerMapping' defined in class path resource [org/springframework/boot/autoconfigure/web/servlet/WebMvcAutoConfiguration$EnableWebMvcConfiguration.class]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Ambiguous mapping. Cannot map 'application2' method 
hello.Application2#home()
to { [/]}: There is already 'application' bean method
hello.Application#home() mapped.
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1804) ~[spring-beans-5.3.21.jar!/:5.3.21]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620) ~[spring-beans-5.3.21.jar!/:5.3.21]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.21.jar!/:5.3.21]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.21.jar!/:5.3.21]


This cause when there are two same Request mapping , in Application.java and in Application2.java.

More precisely,
Application.java
 @SpringBootApplication
@RestController
public class Application {

        @RequestMapping("/")
        public String home() {
                return "Hello Docker World";
        }

        public static void main(String[] args) {
                SpringApplication.run(Application.class, args);
        }

}
Application2.java
 @SpringBootApplication
@RestController
public class Application2 {

        @RequestMapping("/")
        public String home() {
                return "Hello Docker World";
        }


}

com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

Quotation of part of error message
com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
	at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174) ~[mysql-connector-java-8.0.29.jar!/:8.0.29]
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64) ~[mysql-connector-java-8.0.29.jar!/:8.0.29]
	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:828) ~[mysql-connector-java-8.0.29.jar!/:8.0.29]
	at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:448) ~[mysql-connector-java-8.0.29.jar!/:8.0.29]
	at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241) ~[mysql-connector-java-8.0.29.jar!/:8.0.29]
	at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) ~[mysql-connector-java-8.0.29.jar!/:8.0.29]
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-4.0.3.jar!/:na]
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364) ~[HikariCP-4.0.3.jar!/:na]
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) ~[HikariCP-4.0.3.jar!/:na]
 
 /////
 
 Caused by: java.net.UnknownHostException: any: Name or service not known
	at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[na:na]
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929) ~[na:na]
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1529) ~[na:na]
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848) ~[na:na]
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1519) ~[na:na]
The message in above Error ,"The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server." , enable us to guesss that the destination we suppose to send is wrong.
Practically , checking the application.yml , the host name of database server is wrong. application.yml
server:
  port: 8080

#TODO: figure out why I need this here and in bootstrap.yml
spring:
  application:
    name: testLatticeApp

  datasource:
    url: jdbc:mysql://any:3306/sample?useLegacyDatetimeCode=false&serverTimezone=Asia/Tokyo   #"any" is wrong. According to docker-compose.yml , it should be db.
    username: root
    password: root
    driver-class-name: com.mysql.cj.jdbc.Driver


ribbon:
  ServerListRefreshInterval: 1000

endpoints:
  health:
    sensitive: false
  restart:
    enabled: true
  shutdown:
    enabled: true

How to run SpringBoot as docker container

As I struggled to construct the Spring Boot environment with docker ,and successed in it, I write the procedure I tried.

Write a Dockerfile and docker-compose.yml

At first , we prepare a Dockerfile and a docker-compose.yml as follows. 

docker-compose_sample_git.yml
version: "3"
services:
  db:
    image: mysql:8
    container_name: "spring_db_git"
    ports:
      - "3306:3306"
    volumes:
      - ./mysql/sql/:/docker-entrypoint-initdb.d
      - ./mysql/settings/:/var/lib/mysql
      - ./mysql/sql/mysql.cnf:/etc/mysql/conf.d/my.cnf
    environment:
      MYSQL_DATABASE: "sample" # database named "sample" is created
      MYSQL_ROOT_USER: "root" 
      MYSQL_ROOT_PASSWORD: "root"
      TZ: "Asia/Tokyo"


# SpringBootApplication
  spring:
    container_name: "spring_compose_git"
    build:
      context: .
      dockerfile: Dockerfile
    restart: always
    ports:
      - "8080:8080"
    tty: true
    depends_on:
      - db
    volumes:
      - ./spring_git:/srv:cached
    working_dir: /srv
Dockerfile
FROM openjdk:11
RUN apt-get update && apt-get -y upgrade \
 && apt-get install -y vim less wget git unzip \
 && wget https://dlcdn.apache.org/maven/maven-3/3.8.6/binaries/apache-maven-3.8.6-bin.zip \
 && unzip apache-maven-3.8.6-bin.zip 
Then we execute next command in the directory where docker-compose_sample_git.yml and Dockerfile exits.
$docker-compose -f docker-compose_sample_git.yml  up  --build -d

Setting environment more

And then , we login spring_compose_git container and set environmetnt using some more command.
$ docker exec -it spring_compose_git /bin/bash     # get in container
root@01720fff8d82:/srv#       #internal container

Cloning the repository of SpringBoot

# git clone https://github.com/spring-guides/gs-spring-boot-docker.git
# cd / 

Installing maven refering this page 

# wget https://dlcdn.apache.org/maven/maven-3/3.8.6/binaries/apache-maven-3.8.6-bin.zip
# unzip apache-maven-3.8.6-bin.zip

Editing PATH Variable

# export PATH=$PATH:/apache-maven-3.8.6/bin 
# cd /srv/gs-spring-boot-docker/complete

Maven build , skipping test Directory

# ./mvnw package -Dmaven.test.skip 
If you can confirm the message "build success" , you can go next step.

Editting application.yml

Next , we edit application.yml as follows.
# pwd
/srv/gs-spring-boot-docker/complete/src/main/resources
# ls
application.yml
application.yml
server:
  port: 8080

#TODO: figure out why I need this here and in bootstrap.yml
spring:
  application:
    name: testLatticeApp

  datasource:
    url: jdbc:mysql://db:3306/sample?useLegacyDatetimeCode=false&serverTimezone=Asia/Tokyo
    username: root
    password: root
    driver-class-name: com.mysql.cj.jdbc.Driver


ribbon:
  ServerListRefreshInterval: 1000

endpoints:
  health:
    sensitive: false
  restart:
    enabled: true
  shutdown:
    enabled: true


Edittting pom.xml

# pwd
/srv/gs-spring-boot-docker/complete
# ls
Dockerfile  build.gradle  gradle  gradlew  gradlew.bat	mvnw  mvnw.cmd	pom.xml  src  target
pom.xml
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemalocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelversion>4.0.0</modelversion>
        <parent>
                <groupid>org.springframework.boot</groupid>
                <artifactid>spring-boot-starter-parent</artifactid>
                <version>2.7.1</version>
                <relativepath> <!--lookup parent from repository-->
        </relativepath></parent>
        <groupid>com.example</groupid>
        <artifactid>spring-boot-docker-complete</artifactid>
        <version>0.0.1-SNAPSHOT</version>
        <name>spring-boot-docker-complete</name>
        <description>Demo project for Spring Boot</description>
        <properties>
                <java .version="">1.8</java>
        </properties>
        <dependencies>
                <dependency>
                        <groupid>org.springframework.boot</groupid>
                        <artifactid>spring-boot-starter-web</artifactid>
                </dependency>

                <dependency>
                        <groupid>org.springframework.boot</groupid>
                        <artifactid>spring-boot-starter-test</artifactid>
                        <scope>test</scope>
                                                                </dependency>

                <dependency>
                       <groupid>mysql</groupid>
                       <artifactid>mysql-connector-java</artifactid>
                       <scope>runtime</scope>
                </dependency>

                <dependency>
                       <groupid>org.springframework.boot</groupid>
                       <artifactid>spring-boot-starter-data-jdbc</artifactid>
                 </dependency>

        </dependencies>
        <build>
                <plugins>
                        <plugin>
                                <groupid>org.springframework.boot</groupid>
                                <artifactid>spring-boot-maven-plugin</artifactid>
                        </plugin>
                </plugins>
        </build>
</project>


Create a Controller

Furthermore we create new file , TestDB.java , as follows.
# pwd
/srv/gs-spring-boot-docker/complete/src/main/java/hello
# ls
Application.java  TestDB.java
TestDB.java
package hello;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;


import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;


@SpringBootApplication
@RestController
public class TestDB {

        @Autowired
        JdbcTemplate jdbcTemplate;


        @RequestMapping("/db")
        public String testDB() {
                return jdbcTemplate.queryForList("SELECT * FROM sample").toString();
        }



}

Build again

Finally , let's build project again
# pwd
/srv/gs-spring-boot-docker/complete
# ./mvnw package -Dmaven.test.skip 

Insert some data into mysql

Before booting SpringBoot Project , we insert somedata into mysql (spring_db_git container).

we create sample table in sample database ,and insert some data with next command 
 
mysql> create table sample(ID int(10));
Query OK, 0 rows affected, 1 warning (0.09 sec)

mysql> insert sample value(10);
Query OK, 1 row affected (0.04 sec)

mysql> select * from sample;
+------+
| ID   |
+------+
|   10 |
+------+
1 row in set (0.00 sec)

Boot SpringProject and Access

If we succusess in building the Spring Project , let's boot the project.
### check the target directory and which to exectute .jar file.
#java -jar target/spring-boot-docker-complete-0.0.1-SNAPSHOT.jar  
And then , we try to access URL http://localhost:8080/db from browser in host.

If we are in successful in setting the environment, we must see the following message.



[{ID=10}]

2020年4月21日火曜日

Several ways to use Mnist dataset

Introduction

The Mnist hand written digit database is one of the most famous dataset in machine learning.

Although they are maintained in several well known library, as it seems that there are several ways to utilize them and several type of datasets, I confused whether there are something difference.

Because I guess there are anyone like me, I wrote this article to maintain confused information.

Because I already wrote this article in Japanese and referred some reference in that, in this article I'm suppose to show reference in minimum.

Assumption

I assume that you already installed sklearn, tensorflow and pytorch.(Anyway as for me I installed them with Anaconda.)

Furthermore I use MacOSX

Notation

We can see two types of mnist so called hand written dataset.

The first is the one attached to sklearn.

And the second is the others.

The first one is made up of 8×8 pixels.

And the second is 28×28 pixels.

The data attached to sklearn (8×8pixel)

where they are

The dataset attached to sklearn is in the following directory.
/(depending on environment respectively)/lib/python3.7/site-packages/sklearn/datasets
The follow is in my case. (I use Anaconda)
$ls  /Users/hiroshi/opt/anaconda3/lib/python3.7/site-packages/sklearn/    

__check_build   dummy.py   model_selection
__init__.py   ensemble   multiclass.py
__pycache__   exceptions.py   multioutput.py
_build_utils   experimental   naive_bayes.py
_config.py   externals   neighbors
_distributor_init.py  feature_extraction  neural_network
_isotonic.cpython-37m-darwin.so feature_selection  pipeline.py
base.py    gaussian_process  preprocessing
calibration.py   impute    random_projection.py
cluster    inspection   semi_supervised
compose    isotonic.py   setup.py
conftest.py   kernel_approximation.py  svm
covariance   kernel_ridge.py   tests
cross_decomposition  linear_model   tree
datasets   manifold   utils
decomposition   metrics
discriminant_analysis.py mixture
And looking the inside the dataset directory, you might find as follows.
$ls /Users/hiroshi/opt/anaconda3/lib/python3.7/site-packages/sklearn/datasets

__init__.py     california_housing.py
__pycache__     covtype.py
_base.py     data
_california_housing.py    descr
_covtype.py     images
_kddcup99.py     kddcup99.py
_lfw.py      lfw.py
_olivetti_faces.py    olivetti_faces.py
_openml.py     openml.py
_rcv1.py     rcv1.py
_samples_generator.py    samples_generator.py
_species_distributions.py   setup.py
_svmlight_format_fast.cpython-37m-darwin.so species_distributions.py
_svmlight_format_io.py    svmlight_format.py
_twenty_newsgroups.py    tests
base.py      twenty_newsgroups.py
Here you can see the other datasets beside mnist.

And diving into the dataset directory more deeply, you might find as follows.
$ ls /Users/hiroshi/opt/anaconda3/lib/python3.7/site-packages/sklearn/datasets/data
boston_house_prices.csv  diabetes_target.csv.gz  linnerud_exercise.csv
breast_cancer.csv  digits.csv.gz   linnerud_physiological.csv
diabetes_data.csv.gz  iris.csv   wine_data.csv
Here there is some datasets like iris dataset , boston_house_price dataset and so on that are often referred some article about skelearn.

How to import dataset

It is same to official page of sklearn.

The subsequent task is done launching the python from terminal.
>>> from sklearn.datasets import load_digits
>>> import matplotlib.pyplot as plt
>>> digit=load_digits()
>>> digit.data.shape
(1797, 64)     

>>> plt.gray()
>>> digit.images[0]
array([[ 0.,  0.,  5., 13.,  9.,  1.,  0.,  0.],
       [ 0.,  0., 13., 15., 10., 15.,  5.,  0.],
       [ 0.,  3., 15.,  2.,  0., 11.,  8.,  0.],
       [ 0.,  4., 12.,  0.,  0.,  8.,  8.,  0.],
       [ 0.,  5.,  8.,  0.,  0.,  9.,  8.,  0.],
       [ 0.,  4., 11.,  0.,  1., 12.,  7.,  0.],
       [ 0.,  2., 14.,  5., 10., 12.,  0.,  0.],
       [ 0.,  0.,  6., 13., 10.,  0.,  0.,  0.]])
>>> plt.matshow(digit.images[0])
>>> plt.show()

And the following image will appear.



Download the original dataset(28×28pixel)

The original dataset of mnist is in this page.

But the data you can get there is binary data which you cannot use it as it is.

So you need to process them to utilize.

But as you will see , the mnist dataset is so famous dataset that there are a lot of tools to use them immediately.

Of course , although the way to process them by yourself exit, as I couldn't catch up with it and I thought I wondered whether we took much time to seek the way, I'm not suppose to talk about the way.

Download via sklearn(28×28pixel)

Searching internet, in some old article I could find the following way.
from sklearn.datasets import fetch_mldata
But it shows us error , as the website we are suppose to access is not available.

So nowadays it seem that we have to use fetch_openml as follows.
>>> import matplotlib.pyplot as plt  
>>> from sklearn.datasets import fetch_openml
>>> digits = fetch_openml(name='mnist_784', version=1)
>>> digits.data.shape
(70000, 784)
>>> plt.imshow(digits.data[0].reshape(28,28), cmap=plt.cm.gray_r)

>>>>>> plt.show()



tensorflow(28×28pixel)

This is the way using the tutorials of tensorflow.
>>> from tensorflow.examples.tutorials.mnist import input_data
Although this command might enable us to import mnist, it didn't. In my case I faced the following error.

As a result There may be some case where the directory including the tutorial isn't downloaded with tensorflow.

Traceback (most recent call last):
  File "", line 1, in 
ModuleNotFoundError: No module named 'tensorflow.examples.tutorials'
I tried to check inside of directory practically.This is the result.
$ls /Users/hiroshi/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/examples/
__init__.py __pycache__ saved_model
I referred the following pages

At first,you accessgithub page of Tensorflow and download zip file in anywhere and open.



we can find the directory named "tensorflow-master", and in the directory named tensorflow-master\tensorflow\examples\ , there is a directory named "tutorials".

we copy the directory ,"tutorials" into "/Users/hiroshi/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/examples/"

Then,
>>> import matplotlib.pyplot as plt   
>>> from tensorflow.examples.tutorials.mnist import input_data
>>> mnist = input_data.read_data_sets("MNIST_data", one_hot=True)
>>> im = mnist.train.images[1]
>>> im = im.reshape(-1, 28)
>>> plt.imshow(im)

>>> plt.show()

and you can find the image of mnist.

keras(28×28pixel)

>>> import matplotlib.pyplot as plt   
>>> import tensorflow as tf
>>> mnist = tf.keras.datasets.mnist
>>> mnist
>>> mnist_data = mnist.load_data()
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 1s 0us/step
>>> type(mnist_data[0])
   
>>> len(mnist_data[0])
2
>>> len(mnist_data[0][0])
60000
>>> len(mnist_data[0][0][1])
28
>>> mnist_data[0][0][1].shape
(28, 28)

>>> plt.imshow(mnist_data[0][0][1],cmap=plt.cm.gray_r)

>>> plt.show()
I'm not suppose to show image I got, but if you are in success in doing procedure , you must find the image.

pytorch(28×28pixel)

It seems that if you can't run the following command you can't go next.
>>> from torchvision.datasets import MNIST
I faced an error.

It seems that torchvision don't exit.

In my case, when installing pytorch , I merely do as follows. It seems to be the reason.
$conda install pytorch
In order to install some options, you have to do as follows.
$conda install pytorch torchvision -c pytorch 
As you are required to choose y or n, you choose y.

Doing it ( if you need),you run the following command.
>>> import matplotlib.pyplot as plt   


>>> import torchvision.transforms as transforms
>>> from torch.utils.data import DataLoader
>>> from torchvision.datasets import MNIST
>>> mnist_data = MNIST('~/tmp/mnist', train=True, download=True, transform=transforms.ToTensor())
>>> data_loader = DataLoader(mnist_data,batch_size=4,shuffle=False)
>>> data_iter = iter(data_loader)
>>> images, labels = data_iter.next()
>>> npimg = images[0].numpy()
>>> npimg = npimg.reshape((28, 28))
>>> plt.imshow(npimg, cmap='gray')

>>plt.show()
The sources I referred

The original dataset of mnist

sklearn

Tensorflow

The others

2020年2月16日日曜日

What happen if not doing bow rehair

The other day I found my bow's hair couldn't be tighten.

I guessed that it was because I didn't  do bow rehair for long time.

Furthermore the part of bow's frog seem to break , so my bow couldn't adjust how tighten my bow's hair is.


We can see a frog on the frog -1 


We can see a crack on the frog -2





In the case it is expected that my bow will completely break during our performance. As I want to be avoid such a situation, I thought I took measure to my bow.


Going to the repair shop I was said that it is too hard to repair it. And they explained the reason and the structure of the bow.


According to their explanation, my bow's hair has so completely stretched and movable space to adjust how tighten my bow's is has used fully that I can not tighten more.

Moreover , even though in such a case if I turn a screw by force, the screw may break.



They say it takes me as same money as I buy a new one. So I hesitate over whether to buy the new one or to borrow from my acquaintance for time being to get over performance.


Fortunately I could find acquaintance to lend me bow, I decided to use it.


But it is true my bow is so old that I buy a new one after tomorrow's performance.


2020年2月8日土曜日

I took part in the event about English

I'm suppose to talk about this week.

This week was a little busy.


First , as the project I take part in will finish by this month, I was introduce next project and went to take interview with the client.


Though I cannot talk about detail, next one seem to be related RPA.

I hear it seem to be fashionable these days.

I'm not sure whether I'm employed or not , I'm looking forward to join next project.





And, on Friday.

I participated the outdoor event about English.

It is 1000 speakers conference.  I was the first time.


https://1000-speakers.connpass.com/event/163088/



According to this introduction, this event is held to give us opportunity to make presentation in English as much as possible.


This event aim to give 1000 speaker opportunity finally.

As they don't force us to make presentation and it is OK to just listening to.

As I was the new comer , I didn't make presentation.  But totally the atmosphere was so good that I want to take part in next time too , preparing an interesting presentation.


If you want to try to join this , asking me and let's join it together!!

2020年2月2日日曜日

information from our eyes

As current my job is engineer , I may not get so tired at least physically.

But I am always tired.

Although there are a lot of things I have to do , I don't get well motivated.



There may be the opinion that I don't have split enough.  But I don't like such a spiritual discussion.



I agree to use such words , "sprits" or "guts", at last of last.  But there may be rational measure we have to do before it.



I tried to consider why I am always tired.

The first is because my posture is bad.

In the other day , I was advised that my posture was so bad that I seemed not to be confidence.



Checking my posture , I found I work with my posture bad during a day.   And it makes my neck really stiff.


And I found my posture is good when I play instrument.


Although I couldn't have image what is good posture, working with good posture , it made my neck a  little better.





Furthermore is jam-packed train.


Just hearing the word "jam-packed train" makes me so tired.   But let's dive into it.



I feel I get a lot of information while I'm just on the train.


For instance,  enormous people before my face , people's moving occurring every time the train get to the next station , smell , advertisement , landscape outside train , noise made by train , and information from mobile phone that I a little automatically check.


Adding that people adhere to me is also stress.


Moreover , as I'm engineer, I look the display of computer during a day.


A lot of stimulus invoke rambling thought.   So I feel my attention scatter everywhere and all kind of thought run in my mind.


As I thought it is too much information, I tried to close my eye as much as possible.


Closing my eyes and restricting information from external, it make my brain neutral and , if possible , I can be into myself.


consciously or unconsciously reacting the stimulus one by one , I felt I was hurried off to do a lot of thing.


But closing my eye make me relaxed.




It might be one of the mindfulness.


2020年1月25日土曜日

Take it easy

This is the first post of this year.

Last year I already post some article. But since then I couldn't post any one.



I resolved to write any post and tried to consider how I can write article.

Because I have so many task and run another blog in Japanese that I cannot give first priority to writing these blog post.



As the result, taking it easy , I resolved to try write one post a week.
(although I already miss the first post of this year)




From beginning of this year , I review vocabulary and colloquial English.

There is about 800 word and 300 phrase.



As I am reviewing them everyday,  I came to answer in flash.

To repeating reviewing is important.






 That's about it .

I made it a rule to take it easy. So see you next post.


2019年11月9日土曜日

Micro-challenge

Recently I make it a rule to do some small challenge everyday as much as possible, which means that I do something I have never done.


Anyway the company , ATJC , I have entered , hold some club activity, for example ESS , machine learning , Sports and so on.


I have already attended the "machine learning club".

But I tried to attend the ESS club first time.

Though I went to the school for interpreter , because of the extra work, I couldn't go to the school.


So as I haven't spoken English these days, I feel my ability became low a little.

It is the today's first micro-challenge.




After that, as our office locate near the Imperial palace, the enjoyable sound came from outside window.

Though I was suppose to go home soon , but as my colleagues started the small party in our office I also attend it.

I took some picture.







Finishing it , I send these pictures to public relations department to suggest that how about using them when you post an article on official Facebook page.


That I suggested such thing is the today's second challenge.

Because what I used to be isn't such a progressive person.

2019年8月25日日曜日

How I could change the job from the inexperienced

As fortunately I could change my job and the field I'm suppose to work was decided , I try to write about my experience which is , at least in Japan , fashionable.

I'm sorry that it will be very long.



The beginning

Although I went to graduate school where I majored mathematics , namely Number Theory , Modular form, as I fount it is difficult to make a living in mathematical field , I was trying to get job excepting researchers of mathematics.

But I graduated from the graduate school not getting any job.

After many twists and turns , I had been worked as a technical support operator in one of the leading companies for four years.

In other word , I worked in customer support center where I dealt with the inquiry users asked us.

If not mathematics , I wanted to make my way to get academic post of the other field.

But at that time I didn't have enough money and wasn't capable going to graduate school.

So at first I aimed to get engineer's job as a regular employee.

Fortunately and unfortunately the office I had been working for was so comfortable that four years had past.

Having a sense of crisis , I had began job hunting.

I had started job hunting

The first time, later 2016 , the second time , earlier 2018 , I had been doing job hunting.

When 2016 I used agent for job hunting.

There was two reasons why this time I stopped job hunting.

One is that I had lost all of screening the companies at that time I had applied held although I don't remember how many companies I had applied.

Another is that the agent who took charge of me promote my job hunting so fast that I had got a little tired.

Then after a little break , sense of crisis that I had to do job hunting welling up inside me , I had decided to begin job hunting again in earlier 2018.

This time I learned to write resume better than 2016.(But compared with today , it didn't satisfy me.)

But I lost all again.

Policy Change

At that time I had been learning programing and so on by myself , It seemed weak to appeal.

And it seemed that it didn't tell personnel staff how serious I want to get a job as an engineer.

As a result , that I had tried to think about what to do to tell personnel staff how serious I was made me came up with that going to school to learn programing.

Although knowing its exist as I often saw such a advertisement , trying to search eagerly , I found there are a lot of such a school.

My finance wasn't good at that time. But I thought doing nothing occur nothing and if fail, I guessed the fact I had made action might also be appeal.

Deciding to take some break , at first I shifted my policy to go programing school.

Attending the briefing session and taking a screening , finally I got permission from four school and had decided to enter one of them.

Because of curriculum of the school , it was so difficult to continue the job that when I suggested "why don't you continue to work?" I told my boss that I quitted the job.

I had so many paid vacation that use up all my one for one month , July 2018, at the same time I had begun to go the programing school.

The course I went to was the one to aim to be AI engineer or data science engineer which had supports to be it.

It was July 2018.

Going Programing School.

In the school I had some peer and we did team development.

 Learning with our teacher and peer made me realize not only the main contents but also common knowledge , the way to thinking as an engineer , and what is my appeal point and what is my peer's appeal point.

Especially , at that time I had no experiment I could be proud of because I had worked only customer support.

And I , who had been loosing many screening , had thought myself not capable and been pessimistic.

Knowing that I can do easily what others suffering from doing made realize I had something I'm proud of and got confidence.

But the environment was good , as I started to go school without enough plan in budget , I could not help leaving the school.

I went back to square one.

 Above all , I didn't have money so much that first I decided to work as whatever jobs.

 As I had an experience and skill of technical support , I had came to work as the one , though where I had to commute was different from the former.

This was about from November to December ,2018.

Turning Point1

Although I was so busy that I have to stop job hunting, this time I carried on having emotion that I wanted to restart job hunting.

The other hand , though I daren't to talk about its detail , as other many happening made me worn out.

By the way , when I got accustom to the job I decided to work for again,

I got a contact from acquaintance who was my peer at the programing school.

He told that I could get new job.

And he said that as the company his worked for recruited new data scientist , how about applying to it.

 Although so was when I decided to go programing school , I think I got the way to think that when a chance come I try taking concrete action without carefully thinking.

I applied to my acquaintance's offer.

Although I could get to the interview , I've lost the screening.


But this time, as I applied through the employee ,who was my acquaintance , not in general , the interviewer wasn't formal and gave some frank opinion.

As a result I've lost the screening, I thought I could make my way a little because I could understand what levels I was in.

Turning Point2

In the customer center I began to work again , I didn't have much relationship with my peer.

Because , I thought that as I didn't intend to work long term here I wasn't necessary to build deep relation ship with peer.

Although above all I'm not good at building intimate relationship, I had a distance from other than before.

But as my generation was intimate in general , they tried to have good relationship and we got friendly.

One day we went to eat "Monja".

Frankly saying I'm not good at such dishes like I need to cook by myself. Although I hesitated over whether I should go or not , I decided to go.

 I'm not good at cooking "Okonomiyaki", beside I did't understand even how to cook "Monja".

 When I was asked to cook "Monja" , although I hesitated over whether I should take charge of it or not, I decided to try to cook "Monja"

Although I was not sure how to cook "Monja", even if I failed to do it , it wouldn't kill me.

If so, I thought I tried to do it.

And I , who didn't understand how to , put the ingredient of "Monja" into the iron plate at once.

Then I was laughed at by the younger girl, who also took part in there.

But strangely I wasn't embarrassed.

Although I apt not to have common sense, regretting doesn't make me progress.

I had came to think that it was good that I got what I hadn't known.

Although I wrote "I was laughed at by the younger", I think it was not "she gave me a smile" but "I was laughed at by her".
But as she was charming , any response of her made me happy.

 It caused me strangely to assemble the structure that
"I was laughed at "
= "I'm doing what is not common in public"
= "I'm just ignorant or I'm doing something".

And I was getting comfortable to be laughed at.

If it was not "they give me smiles" but "I was laughed at by them" , so is it.

And the motto "Enjoy failing" was generate in myself.

I resumed job hunting

 I had met the acquaintance , who I told above, the peer in the programing school, again, and I report the interview I had tried the other day.

 We also talked about that unexpectedly we could find recruitment if it was OK for me to be temporary staff.

 It caused me to register lots and lots web-site for job hunting and apply lots and lots company , following the sprits "Enjoy failing"

 We can see web-site for job hunting so much and it was so difficult to complete my resume that I wrote and improve my resume slowly everyday.

 Furthermore I applied to the company I was interested in even a little, "automatically" and "making it a rule that I had to apply two or three companies a day."

 I think I applied to 40 ~ 50 companies until I got promise of employment in the end.

I was crushed my expectation.

I could through the screening of one of these companies, where I was scheduled to do final interview.

As It seemed that I could gave good impression in the screening of resume, I was let to do second interview in the same day.

But when the schedule I had to do was only final interview with the president of the company, they assigned me coding task not written in flow of screening steps as advance notice.

 I didn't set deadline.

 Although I did the assignment in about one week , I could't finish doing it.
I postponed the deadline.

Finally I couldn't complete it , as I hesitated over whether it was O.K. I let them wait, I decided to submit it. 

But I had lost screening.

 As I could though first and second interview smoothly and what I was scheduled was only final interview, I was disappointed.

But I was told that they permit I posses the copyright and if it was necessary when I applied to other company I could use it.
So I decided to use it without any hesitation.

Promise of employment

 Continuously I had been applying many companies making a norma that I had to apply two or three companies.

Although I often lost the screening of resume, I could also get to the interview.

At that time , I lived a life that I went to do interview on my holiday (it was weekday in public) working for customer center 5days in a week.

If I call on 1 company, it costs me expense to move and takes me several hours including the time to move.

So at that time , if I was asked to come to the company to do interview, I think , I asked them to loose me in the step of screening of resume if they have little expectation to through interview.

 While living a such life , surprisingly I got promise of employment at the end of April !!

Then I stop applying newly.

I decided to finish job hunting after I confirmed the result I was waiting at that time.

More surprisingly I got another promise.

Although I'm not sure now ,it might have been effective to tell that I had already had a promise.

Is the trump card that we tell "I already have some promise of employ" strong?

Although the salary of the first company was better than second, considering my career , atmosphere and so on, I decided to enter the second company.

Persuasion

It was happy for me that I was also persuaded by the first company.

They told me that they hadn't been able to tell me how attractive their company , could I set the interview to talk about the matter again.

Under the condition that I didn't set any premise to change my intention , I met them again.

This time it wasn't official interview , we could talk frankly so much.

 Their atmosphere wasn't bad. But the second interview made me more sure that another company seemed to be suitable for me, I decided to decline to enter the company.

Training

Although I guess the training of any company is same, the training was the style that we answer the question given as training.

I learned about Java.

The personnel staff and teacher , who know how skillful I am checking my resume , portfolio , told me that finish all task in half a term other trainee was given.

Partly because I couldn't remove some bugs , partly because it was unexpectedly bulky , in the meantime I finished the task almost the same time.

But as I had richer knowledge than my peer , I often taught to them.

The other hand I was so skill that the company seemed not to delivery for boring site.
Because it seemed that they considered where to deliver me.

Finally finishing the training , the site I was suppose to work was decided.

Taking up my first post

 I took up the first post where I was suppose to use Java and Spring boot.

 I could take charge of from the step of detail design. As I have been doing programing by myself and decide the name of variable as I like, I'm struggling to make specification for screen by excel.

Reflection

If I try to reflect on these process, I think I was important to make rich my resume or portfolio so as to change jobs from beginner.

In my case I referred to the resume and document of work experience the acquaintance , who was my peer and I told about above, made

Referring them , I found them so rich I learned how to write.

The result I described on web-site for job hunting or document of work experience was :

  • URL of github (I made its repositories tidy)
  • The assignment given by the company I told in my sentense.
  • This Blog (I also write this blog in Japanese and they are more rich) 
 and so on.


Moreover , although I magnified document of work experience a little , I described everything that even if I had use them a little.

It is important to speak keyword in a interview , to be sure , because the words they use show us how skillful they are, but if showing the code they write we can judge their skill more precisely.

 Furthermore , although I'm afraid of that if may be a little lengthly, that fact that I could make the motto "Enjoy failing" is a critical factor.

Whether I told myself the phrase in imperative form or in active form depends on my feeling. But I think that it seems that when I face what I hesitate , repeating this phrase in my mind had me get courage.

Tracing the process I reach there , the original cause was the event I went to eat "Monja" with my peer at that time.

It might be just a "Monja".

But such a critical event might be concealed in routine life unexpectedly.

2019年3月4日月曜日

How to run Perl script

I came to know that to start using Perl seemed to be so easy as to build the environment of Perl.

I suppose to keep a record how I built in order that I can remember afterward.



The script we are suppose to run

This time's goal is to run the following script.

Writing the following script on text editor , save it as "hello.pl" .

#! /usr/local/bin/perl

print "HelloWorld!!\n" ;

Confirm the version of Perl

First ,In any directory , run the follow script.
$ perl -v
Then you can see such like message.
The following is the one in my case.
$ perl -v
This is perl 5, version 28, subversion 0 (v5.28.0) built for darwin-thread-multi-2level

Copyright 1987-2018, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

Where the Perl is installed

Next we confirm where the Perl is installed.

Running the following command , we can see some message like the following.

The following is my case.
$which perl
/usr/local/bin/perl

Where to put on the script we wrote

Without any your preference , making new directory under the home directory as perl , you had better put the program in there.

If you want to do so, you run the following command , and put the program in there.
$cd
$mkdir perl

Through the Path

we set the environment variable.

Run the following command.

$PATH=~/perl:"$PATH"
In order to confirm whether we could set the environment variable correctly, you had better next command.
$echo PATH
In the message you can see, if you see the sequence of letter you set , you are successful in it.

Modify the Permission

We have to give the authority to perl program.

Run the following command.
$chmod a+x hello.pl

Run!!

Anyway if you doing like above , you can already run the script.

Moving the ~/perl , and run the script.
$cd perl
$hello.pl
You can see
HelloWorld!!



Although I wrote such a article , because I'm not good at English enough, I omitted a lot of what I want to write.
I wish I could write more fluently....
I'm suppose to improve myself more!!

2018年9月28日金曜日

My interest

Until the other day, I went to the cram school to be a machine learning engineer , quiting  a job.

Though I halt the going to the school because of the budget , I try to talk about AI as who learn machine learning somewhat.

From before I decided to go the school , I have learned the AI or machine learning.

But I didn't write any program and understand strictly mathematically.

I , who have written some program of machine learning , can talk about what I'm interesting better than ever.

What I felt that the program we don't or haven't been able to write is "being convinced (deep understanding)" , "motivation" and so on.

As for us , humanity , we haven't understand the difference between "simple understanding" and "deep understanding".

Have you experienced the condition that you yourself don't know whether you understand distinctly or not or that you can not agree emotionally with what the other person say though you can understand it rationally.

The difference between "simple" and "deep" is above.

As for "motivation", it is same.

I felt that we haven't written the code of them.

Can we write such programs which express them?

2018年9月27日木曜日

Anxiety

This is my first post in English.

Though I have been writing blog in Japanese, as I think it is better to let everybody in the world what I'm thinking about , I decide to try to write some post.

Anyway the contents I want to write is my troubles.

Sometimes there is times when I'm not suitable for researcher, though I wanted to be and aim to be.

This mean , recently I don't have adequate money to live comfortable life.

In order to raise my income, I invest myself , for example English conversation or engineering skill for AI the other world machin learning.

Even though there are many what I have to learn, the shortage makes me so melancholy that I can not concentrate on studying.

Is this mean I'm not suitable for researcher?

In my Image researcher fight many pressure.  In case I become a researcher, could I endure such a pressure?


That's all I want to write first.

Because I'm studying English there must be many mistakes on grammar or expression.
If you find any mistakes and are kind person, if advising me it I'm very grad.