First, start gdbserver from the device, 3230 is any port number that is not used by the network.
gdbserver localhost:3230 ./helloworld
Second, start arm-linux-debug from the host,
arm-linux-gdb ./helloworld
target remote 10.10.10.20:3230
b main
c
Here b identify set break point, and c identify continue, they are gdb commands.
Right now, I am debugging without emacs, still hope to configure emacs to support embedded debugging.